My previous Instagram data collection approach started failing in ways that were difficult to ignore.
Requests that had worked earlier began returning errors. Some sessions stopped working, data collection became inconsistent, and repeated requests increased the likelihood of temporary restrictions. What should have been a straightforward data collection task turned into an exercise in managing Instagram's access limitations.
The objective was simple: monitor what competitor brands publish on Instagram and understand how their engagement changes over time.
I needed a more predictable way to collect public Instagram data without making my own infrastructure responsible for every authentication and access problem.
That led me to HikerAPI.
What I Was Trying to Build
The goal was to create a competitor monitoring workflow that could collect and organize publicly accessible Instagram information.
The main things I wanted to track were:
Competitor brand usernames and profile information.
Recently published posts.
Captions and publication timestamps.
Likes and comment counts.
Changes in engagement over time.
Posting frequency and content patterns.
The interesting part was not collecting a single post. It was collecting information consistently enough to compare activity across different periods.
For example, knowing that a competitor published a post is useful. Knowing how its engagement changed over the following days is considerably more useful.
Why My Previous Approach Started Breaking
Instagram does not provide unrestricted access to its platform data through ordinary requests.
When collecting data through unofficial scraping methods, several problems become relevant.
- Rate limits
Repeated requests can trigger temporary restrictions. Collecting profiles, posts, followers and engagement information can require multiple requests, making large collection jobs particularly difficult.
- Session and authentication problems
Unofficial scraping libraries often depend on authenticated sessions. Sessions can expire, authentication challenges can appear, and previously working scripts can stop functioning.
- Unpredictable failures
A collection process that works today might fail tomorrow. This makes scheduled monitoring unreliable.
- Maintenance overhead
Instead of concentrating on data processing and competitor analysis, I was spending time dealing with request failures, authentication and changes in Instagram's internal endpoints.
I needed to reduce that operational overhead.
Why I Moved to HikerAPI
I started looking at hosted Instagram data APIs instead of maintaining the entire scraping infrastructure myself.
HikerAPI provides a hosted REST API for accessing Instagram data.
The main difference is that I can make API requests to a dedicated service rather than implementing and maintaining all the underlying collection logic myself.
Some things that attracted me to it:
REST-based requests.
Structured JSON responses.
Hosted infrastructure.
Access to different Instagram data endpoints.
Usage-based pricing.
A free allocation of 100 requests.
Pricing starting from $0.001 per request.
For a small monitoring workflow, the ability to test with a limited number of requests before committing to a larger usage plan was useful.
I also found its documentation on HikerAPI's approach to Instagram data collection useful when evaluating the alternative.
Making a Request with Python
One of the advantages of using a REST API is that the Python implementation is relatively straightforward.
I can use the requests library to communicate with the service.
Here is a basic example of retrieving a user profile and requesting follower data:
import requests
headers = {"x-access-key": "YOUR_KEY"}
user = requests.get(
"https://api.hikerapi.com/v2/user/by/username?username=natgeo",
headers=headers
).json()
r = requests.get(
"https://api.hikerapi.com/gql/user/followers/chunk",
params={"user_id": user["pk"]},
headers=headers
)
print(r.json())
The first request retrieves the user information using a username.
The second request uses the returned profile ID to request follower information.
The API returns JSON, which makes it easier to process the results using Python.
For a production implementation, I would also add:
HTTP status-code validation.
Request timeouts.
Exception handling.
Pagination handling.
Logging.
Secure API-key storage.
I would not treat a successful HTTP response as proof that the returned data is complete. Pagination and API response validation still matter.
Applying This to Competitor Monitoring
The actual value comes from what happens after collecting the data.
I would organize the workflow into four stages.
- Collect profile information
Maintain a list of competitor brand usernames and retrieve their publicly available profile information.
Store the platform's user ID alongside the username so that subsequent requests can reference the correct account.
- Collect recent posts
Retrieve publicly accessible posts and store relevant fields:
Post ID.
Username.
Caption.
Publication timestamp.
Like count.
Comment count.
Media type.
The post ID is particularly important because it allows me to associate subsequent engagement observations with the same post.
- Track engagement changes
Rather than overwriting engagement numbers every time I collect data, I would maintain historical snapshots.
A simple database structure could look like this:
post_id
collected_at
likes
comments
12345
Day 1
120
12
12345
Day 2
165
18
12345
Day 3
210
24
This makes it possible to calculate changes over time.
For example:
def engagement_change(old, new):
if old == 0:
return None
return ((new - old) / old) * 100 likes_change = engagement_change(120, 210)
print(f"Likes increased by {likes_change:.2f}%")
Output:
Likes increased by 75.00%
These numbers are illustrative, not actual competitor measurements.
- Analyze the results
Once the information is stored, Python libraries such as Pandas and Matplotlib can help identify:
Which brands publish most frequently.
Which content formats receive more engagement.
How engagement changes after publication.
Which posting periods correlate with higher engagement.
How competitor activity changes over time.
This is where the project becomes more than a scraper. It becomes a monitoring and analysis system.
What I Gained and What I Didn't
Moving to a hosted API does not eliminate every problem.
There are still tradeoffs.
What improved:
Less responsibility for maintaining scraping infrastructure.
A straightforward REST interface.
Easier integration with Python.
Structured responses that can be stored and analyzed.
A pricing model that can be evaluated against collection requirements.
What still needs attention:
API usage costs can increase with collection frequency.
Rate limits and service restrictions can still exist.
API availability depends on the provider.
Data availability and completeness are not guaranteed.
Instagram can change what information is accessible.
API access does not grant permission to retrieve private-account content.
I also need to consider Instagram's terms, applicable privacy requirements and the API provider's usage policies.
A hosted API changes where the collection infrastructure runs. It does not remove the need to design responsible data collection.
Final Thoughts
The biggest lesson from this process was that collecting data and maintaining a reliable data collection system are two different engineering problems.
Initially, I was spending too much time trying to keep the collection mechanism working. For a competitor monitoring project, the more important work is storing observations, tracking changes and extracting useful information.
Moving the collection layer to a hosted REST API gave me a different architecture to work with.
It is not a guarantee against restrictions or failures, and it introduces a dependency on an external provider. However, for a project focused on monitoring publicly available competitor content and engagement, it is an option worth evaluating.
I would rather spend development time understanding engagement trends than repeatedly debugging a collection script.