Browse AI

Web scraping allows you to extract data from a website and turn it into a neatly organized format like CSV or JSON. You can use it to extract contact information, pricing data, really anything that's displayed on a website. And it's especially useful for very large data sets where you don't want to go and write down everything by hand. In the past, you needed programming skills to do that because you had to tell the computer exactly what HTML element it has to click, where it can find the data to extract, how it can navigate to the next page. And as you can imagine, this was very difficult. But now we have AI that can do that for us. Today, I'm going to show you a tool where you can teach robots to scrape any website. And it's so easy that really anyone can do it. I will walk you through everything step by step. Together, we will scrape a graphics card shop. We will get an email notification with the latest data and see if any prices have changed. And this will work even for pages where we have to log in. My name is Florian Walther and this is the AI tool corner where I review the latest AI software to find out which ones can actually improve our lives and businesses. The tool is called Browse AI. I will put a link into the video description below, and you get 50 free credits every month and an unlimited number of robots, which is actually plenty. This is more than enough to try this out on different websites. Now, web scraping is legal, but some websites don't want you to do this. So, they have different measures like CAPTCHAs and bot detection that block your web scraper. Browse AI has built-in bot evasion, which means that on most websites, this will still work because the AI will pretend to be human. It will resolve CAPTCHAs and I think it even uses different IP addresses to not get blocked. Some websites will have written in their terms and conditions that you are not allowed to scrape them. And if you do it or not is your own decision. I can't condone it. I will not tell you to scrape any websites that don't allow it. But of course, I'm not your mommy and you have to make your own decisions. For the purpose of this tutorial, I set up this fake graphics card shop. I actually vibe coded this with Lovable. I also tried out Browse AI on a real shop and it worked flawlessly. It even worked on large websites with sophisticated bot protection. But I don't want to get this video flagged. So, we will use this fake shop here. So, follow the link in the video description and then create an account on Browse AI. Step one is to get the link for the website we want to scrape. So here over in my shop, I could use the link as it is, but I can also apply some filters like I only want to show a graphics card in the high-end and mid-range categories. We have two pages here. I'm going to copy this link here in Browse AI. We go to robots and then we click one of these options here. Doesn't matter. They both point to the same thing basically. So I'm going to click on Monitor Site changes and here we simply paste the URL. This website does not need login. We will scrape a page with login later, but this one doesn't need login. Then we click on start training robot. And here we select the robot studio option. So we don't have to install a browser extension. And then we get to this virtual browser where we can teach the robot how to extract the data. And this is very easy. Do we want to capture text or a screenshot? We want to capture text. Then we have to select if it's normal text or if it's a list. This is a list of graphics cards, right? So, we select from a list and then we just take our mouse and we hover over the list and as you can see, every item here has this border around it. Now, we click on an item, it recognizes the list and now the AI finds and extracts useful data automatically. And voila, wasn't this easy? The AI even named each column. So, it extracted the product name, the rating, the length, everything that could be important. Now, we can customize this by removing columns. For example, I don't care about the review count or the rating. I basically just want the price and the name of the graphics card. So, you can remove all the columns you don't need. Then, we tell it how many rows we want to extract. For this tutorial, I'm just going to extract the first 10 results, but you can extract more. And then, important, we have to tell it how it can find the next page. Because our data set here is paginated. It only shows up to nine results on one page and then we have to click the next page button. The AI actually recognizes this automatically in most cases. So when I click on review pagination settings, yeah, it detected the number two as the next page button. But it would actually be better if it clicked this button here to get to the next page because then it will also work when we have three or more pages. Right? So up here where it asks us to review the pagination settings, we can click on reselect. Then we select click on next to navigate to the next page because this is the type of pagination we have here, right? And then we simply select the button where it can navigate to the next page. And just like that, our bot can navigate between pages. When we have all of this configured, we click on save and then on finish. We can give this robot a name. It automatically generated one. And then we can tell it how often we want to run this automatically. Let's say we want to extract the graphics card data once every 12 hours, every day. And we want to get notified via email if there is a change in the data that we are parsing. Save. Below we can see the extracted data. And this way we know that this actually worked. If this data is correct, click down here on yes, looks good to save this robot. Here it suggests to train another robot to extract the details for each graphics card. So we can connect two robots together. We will do this later. But for now, what I want to do is I'm going to change some of the pricing on this page. Then we run the robot again to scrape the newest data and see the changes in our table because we want to know how this looks, right? So I changed some prices here and removed one of these cards. And then over in our robot, we have a monitor set up that runs every 12 hours, right? But I don't want to wait 12 hours right now. So we can also click on run a task to rerun it immediately. So this finished successfully. And when we click on history, we can see all the past data. This here is the last one. And when we check this switch here, we can see the changes since the last time we scraped this list. So this graphics card has gotten a bit cheaper. When we remove an item, we will also see this here. If we want to use this data for anything, we can download it as a CSV or JSON file. And our robot will also go ahead and parse the site automatically every 12 hours. And then we actually get an email with the changes. If you want to run this more often or less often, of course, you can update this. And you can also update your robot here in the settings. How many items do you want to parse from what URL? You can retrain your robot to change how it parses the data. But one other useful feature is deep scraping. So here we have the suggestion to train another robot to extract the detail information for each graphics card. Let's click on train another robot. Again we are in our virtual browser, but now we are on a details page of a graphics card. So the AI recognized that each entry in our graphics card list has a link to a details page. And now we can also parse this details page to get information out of here. Again, we can capture text, but this time it's not a list. It's just simple text. And here we can select whatever we want to parse from this page. For example, we want to parse the reviews. Here we select visible text because this is what we want to extract. And let's say we also want the technical specifications. Again, visible text, and select whatever else you want to pull. When you are done, click on confirm. Then we have to give each element a name. So this is the ratings and those are the specs. This is the data we extract from this page. Save. And when we are done, we finish this robot as well. Let's call it GPU product details. Again, we can run this automatically. And again, we save this down here with yes, looks good. But we are not done yet. So now we have two different robots and we can connect them via a workflow. So here we can say whenever our first robot extracts a list of product links automatically run the second robot for all these links and extract the details. So the second robot runs 10 times in parallel. Let's try this out. I'm going to run the first robot again to extract a new list which should then automatically trigger the second robot. So our first robot has finished. It extracted the same nine graphics cards. Let's take a look at our other robot which should now have run nine times in parallel once for each graphics card that we extracted. Again we can look into the history. We see this bulk run because this ran nine times at once. And here we can now see each extracted graphics card with the data that we are parsing from this details page or we can view it as a table where we can see all the data at once. So those are the nine graphics cards with the details. And again, we can extract this as a CSV or JSON file. You can also run this as a bulk task manually, which just means that you give it the list of details, page links to extract, and this can be up to 50,000, which is useful to extract large data sets. Browse AI also has different integrations so that you can sync your data to external apps. For example, you can sync your data into a Google sheet. You can set up a web hook which is useful for developers here. When one of your robots finishes, you can call one of your own server endpoint URLs. But if you're not a developer, then you can ignore this. You can also turn the data that you scrape into an API. Again, this is useful for developers because then the apps can automatically pull in the data that you scraped to use it inside the app. I also want to show you how scraping works with user login. So here in my shop I have a sign-in page and when I log into my account I get access to this favorites page where I have favorite articles. Now let's say we want our robot to scrape the latest prices for our favorited GPUs. Then the robot first has to log into our account to get access to these favorites. Right? So let's see how we can do that. We create a new robot. The page we want to scrape is now our website /favorites, right? So let's paste it here. But this page requires login with a session cookie or with password. I select password here, but both should work. And again, we train this robot. Now in this virtual browser, we log in as usual. And the robot will remember our credentials. Now these credentials will be stored securely on their servers and they will be encrypted. But of course, you have to decide for yourself if you trust this or not. I entered my credentials and down here, you can see it tracks all the page interactions, including what we type in. And it recognized that this is the password. So, it doesn't even show it here in plain text, which is good. And when we click on sign in, we have to navigate back to the favorites page because this is where we want to extract the data. And then again, we want to capture text from a list, which are these three items here. Again we customize our table. Select how many items we want to parse. Configure pagination settings if necessary. Then we save this and click on finish. We give this a name and save it again. Configure how often we want to run this automatically. And now the bot can automatically extract data from inside our account and keep us updated if any of these prices change. If you want to try out Browse AI for free, the link will be in the video description. Please subscribe to the channel for more AI tool reviews in the future. Then I hope we see us in the next video. Take care.