💼 LinkedIn Web Scraping – Scrape Profiles, Groups & Jobs Data with JUST a Browser for Free
LinkedIn Data Collection Tool: https://stevesie.com/apps/linkedin-api LinkedIn – the only platform where any person, for any reason… can complain to a manager. Just look, I managed to find almost 100 million managers I can complain to about something! But that would take forever! I can’t possibly complain to all of them, so we need to filter […]
LinkedIn Data Collection Tool: https://stevesie.com/apps/linkedin-api
LinkedIn – the only platform where any person, for any reason… can complain to a manager.
Just look, I managed to find almost 100 million managers I can complain to about something!
But that would take forever! I can’t possibly complain to all of them, so we need to filter down our list to find the right people.
Let’s try looking for restaurant managers… that gets us down to 2 million, but what if I want to narrow down even further?
It would help if we could scrape & export this list of managers into our own systems & spreadsheets so we can keep a record of them and filter further by location, title, past positions, employers and connection level.
Well if you watch until the end, I’ll show you how to extract raw, structured data from LinkedIn using nothing but your browser with a free tool I offer.
So let’s go back to looking for some managers… I’m going to use LinkedIn’s search feature here and maybe add a few more filters.
Once I’m happy and want to begin scraping the data, I simply need to do 1 thing which will “record” my web traffic. This will have my web browser save all of the HTTP request data (containing the structured LinkedIn data) sent to my browser.
I just right click anywhere on the page and click “Inspect” – this will open up the browser developer tools which will have the browser record & save all of the data you send and receive.
Now we just keep scrolling and paginating through the search results here, knowing that our browser is saving the raw data that LinkedIn is sending us.
Now, I understand this is pretty tedious having to scroll and click all day – but due to LinkedIn’s Terms of Service, if you *were* to *hypothetically* automate this part (say with a Chrome extension), it *may* be a violation of the terms of service… so all I can suggest here is to just manually use the site while recording the data they give you.
Once you’re done scrolling, we now just need to save this raw data to our computer on something called a HAR file. We just click the “Network” tab and “Export HAR…” and now we have the profile information LinkedIn sent us.
Only problem is, that this HAR file has ALL the HTTP requests sent back to the browser with a lot of irrelevant data in it. So we’re going to use the free tool I have linked to below which will process your HAR file and only show you where the structured data is, grouping together similar requests and allowing you to search within the responses.
Free Tool: https://stevesie.com/har-file-web-scraper
And because HAR files contain sensitive data in them, like cookies, this tool runs only in your browser so nothing leaves your computer.
So I’ll just drag and drop the HAR file, and let’s search for the word “Manager” to make sure we find the correct group of network requests.
So here I can see all of those search requests we made, grouped together here. I can download each one to my computer now and see each individual parts of the search results here in JSON format.
And it’s not just search results you can capture from LinkedIn – you can browse and scroll around different parts like group members, jobs, company info, etc… and use this tool to find the data you’re looking for. Simply drag and drop whatever HAR file you generate.
So while these individual JSON files are great, you may not have the time or resources to process them into something useful. E.g. you may need all of these individual search results combined into a single CSV file, so you could track the data in Google Sheets for example.
Well, the Stevesie data platform has you covered – if you need the results in a more usable CSV format, premium Stevesie users will see this “Parse” button here they can use that will upload the JSON from the HAR file and convert it into combined collections you can then download in CSV format.
So here the LinkedIn JSON is a little messy and resulted in a lot of different collections, but if we just scroll through them under “All Collections” here, we can find what looks like individual search results here on this collection. So we can just download these in Expanded CSV format and we’ll have our list of managers now in CSV format.
You can find my work in the link in the description… this way you can simply browse LinkedIn, export your HAR file, then upload to the tool and it will only show you LinkedIn data, allowing you to search through it and download the JSON responses for free, no signup needed!
Just a quick disclaimer though – that accessing the endpoints we found outside of an official LinkedIn client, like their app or website, *might* be against their terms of service and even considered trespass, even though the official client was just going to access the endpoint anyway. It’s a very grey legal area and I don’t suggest you try your luck with it.