What Is ScraProxy

ScraProxy is a super neat project developed by GITHUB_NAME that allows developers to effectively rotate proxy IP's on each request, without going through the hassle of writing fragile logic functionality or internal services.

Traditionally, you'd need to import a list of proxies in the form of a txt file and track which proxy IP address last served the previous request. ScraProxy removes this headache for you, by creating a single, super proxy that you can pass, with it's credentials as an ENV var at runtime.

‍

At a glance, the main benefits of using ScraProxy are: 

• Ban minimization - Smartly rotate requests through different proxy IP's to minimize ban's.
• Request Success Rate - Automatically remove poorly performing proxies from your super proxy pool when a proxy repeatedly fails.
• Infrastructure simplification - You're offered a plethora of hosting choices, and configuration options.
• Intuitive - The fundamental reason why ScraProxy was created is a tail as old as time and something most developers find relatable.

‍

Deployment Walkthrough: 

We're going to do a quick deployment tutorial using the Docker option to understand the powerful features you can leverage when using Stat Proxies residential static proxy infrastructure. We're going to deploy this in a developer environment/ locally, so we're going to leave out the flag: NODE_ENV=production that will reduce the volume of logs generated by the container.

Upon successful deployment, the ScraProxy service will be deployed on port 8890, which you can navigate to by going to localhost:8890. You'll be greeted by a login page, prompting you for the username and password we set at the docker container deployment. In this case, our username is "admin" and our password is "password". _{(please for the love of god do not use these credentials in a production level deployment 🙏)}

‍

If all goes well, you'll see a new project page when you first login. Before getting too overwhelmed, lets go step by step, and understand what we're being asked here.

• Name: The user defined project name. Set this to whatever you want
• Username: The authentication username required for proxy authentication when making requests.
• Password: The authentication password required for proxy authentication when making requests.
• Renew token: A button to click for renewing the username and password.
• Minimum proxies: The minimum number of proxies that should be online when the project status is set to CALM.
• Auto Rotate Proxies: An option to enable automatic rotation of proxies at random intervals within a specified delay range.
• Auto Scale Up: An option to automatically switch the project status to HOT and start all proxies upon receiving a request.
• Auto Scale Down: An option to automatically switch the project status to CALM and stop all proxies if no requests are received after a specified delay.
• *Intercept HTTPS requests with MITM: An option to enable Scrapoxy to intercept and modify HTTPS requests and responses.
• *Certificate: A CA certificate to install to avoid security warnings in browsers or scrapers.
• *Keep the same proxy with cookie injection: An option to enable Scrapoxy to inject a cookie to maintain the same proxy for a browser session (sticky cookie).
• *Override User-Agent: An option to enable Scrapoxy to override the User-Agent header with a value assigned to a proxy instance, ensuring all requests made with that instance have the same User-Agent header.

Note that keys with a * prepended to its username will require you to install a CA certificate on the client device using the ScraProxy Super Proxy.

‍

Lets continue to the next step, setting our our provider. Since we're going to be using our Stat Proxies Static Residential Proxy list, we'll choose the Proxy List option. Click "Create" and you'll be prompted with a few more configuration options

‍

‍

You can prompted to set a name for this provider's credentials. Since we're using a predefined list of proxies, our credentials here are not going to be used. However, let's call it "Stat Proxy List"

‍

After creating our pseudo credentials we'll be brought to a new connector page. This new connector page will prompt us with a few different key options that we'll need to set. In our case, we'll set the credentials as the Stat Proxy List we set in the step before.

‍

‍

• Credential: The credential to use for the connector, selected from the list of available credentials.
• Name: A unique identifier for the connector within the project.
• # of proxies: The maximum number of proxies that the connector can provide and that you intend to use.
• Proxies Timeout: The maximum duration for connecting to a proxy before considering it as offline.
• Proxies Kick: An option to enable the removal of a proxy from the pool if it remains offline for a duration exceeding the specified value. This value must be greater than the Proxies Timeout.

After saving the settings, Scrapoxy performs a validation test to ensure the entered configuration is valid. The list of configured connectors is then displayed. Now, we'll need to go head over to our Stat Proxies client dashboard to grab our proxy list. In this case we're using a 25 pack of Stat Captcha Proxies _{(they are the best datacenter captcha proxies with the highest Google ReCaptcha score :-). )}

‍

Prior to proceeding, we'll be making some modifications to this Proxy List Connector, so the connector must be disabled / off. Make sure that your connector looks like the image above.

We can make the desired modifications by clicking the configuration option, represented by the 3 vertical buttons on the right hand side of the Stat Proxy List Connector. Continue by clicking the "Update" option in the drop down. We'll want to scroll down to the Proxies section. Specifically, we'll want to copy paste over the Stat Captcha Proxy List from the dashboard into the "Add new proxies" text area. We can ignore our sources since this list is static (does not change).

‍

Push the plus ("+") button adjacent to the proxy list text area, which will create a table with all of our proxies. ScraProxy will validate that all the proxies we've set are active, and will prompt us to remove any inactvie proxies automatically with a single power button option.

‍

Once we've added, appended, and validated our proxies, we can click the "Back" button to go back to the Connector List Page. We're now FINALLY 🥳 ready to flip on our ScraProxy Super Proxy. Flip the switch to on. and we can head on over to our Settings Page to grab our credentials.

By Default, ScraProxy provides as username:password authentication mechanism to route traffic through our Stat Captcha Proxies. the Docker container will also have port 8888 open as the proxy port. In effect, this means our ScraProxy will look something along the lines of localhost:8888:u9eww37vz5jqnhauz7zxgp:7ca2kuhk9prpd64ypggtj where: 

• localhost: is our reverse proxy IP
• 8888: is our proxy port
• u9eww37vz5jqnhauz7zxgp: is our reverse proxy username
• 7ca2kuhk9prpd64ypggtj: is our reverse proxy password

‍

After a little bit of time, ScraProxy will populate both the Proxy side nav bar page, along with the Coverage page (what we're looking at in the image above). Here we can check out some high level statistics about our proxy infrastructure.

‍

Concluding Thoughts

‍

There are so many more cool features that you can explore with ScraProxy: From the rich API controls, to the MITM configuration, and even mixing different proxy providers. But we'll stop ourselves here. If you're interested in combining the power of ScraProxy with Stat Proxies Static Residential ISP Infrastructure, pick up a pack today, and take a closer read at ScraProxy's documentation

‍

‍

‍

‍