Project: codeshite/shite comparison/compare the shite

[moosemansjukebox]

 So after a few beers last week, I was thinking, why isn't there a comparison site for used cars? Something that can aggregate the results from the likes of Autotrader, ebay, gumtree, fb marketplace etc. I think there's probably stuff out there for new cars, but not for the stuff i'm interested in (sub 2k shite). I find myself jumping between browser tabs and saved bookmarks. 

So after a few evenings of hacking together some extremely shoddy code, i've built a proof-of-concept for the idea here: https://autoscraper.co.uk

So far I've got something working that can search autotrader, gumtree, ebay and AA used cars. Motors.co.uk is my next challenge then Facebook marketplace...

It's absolutely unfinished, there's no mobile support, the order of the results is confusing, the search is limited and the name makes zero sense 😂

Feedback, ideas, suggestions, criticisms, contributions absolutely welcome! I'm not really sure where i'm going with this,  i'm just having some fun learning some new stuff about web scrapers. I'll post updates as I do them. 

[UltraWomble]

Very good - interetsing to see the same car at different prices on different sites 

[AnnoyingPentium]

Very cool. Could you up the distance limit to national/1000 miles at all? It only goes to 500 for me. Just as a suggestion.

[Missy Charm]

I wish to make a complaint.  Your supposed 'scraper', whatever that may be, is inherently sexist and, as a result, needs to be re-educated or terminated.  I attempted to look for a car under £2,000 in my area and the only result was this stereotypical girls' thing.  Keen observers will also note that the car is broken, presumably because the 'scraper' believes women shouldn't drive at all so is attempting to keep them off the road by displaying adverts only for inoperative vehicles.  This was not, and I repeat not, my fault for accidentally selecting 'Abarth' from the drop down list whilst it was loading!

[moosemansjukebox]

5 hours ago, AnnoyingPentium said:

Very cool. Could you up the distance limit to national/1000 miles at all? It only goes to 500 for me. Just as a suggestion.

Good suggestion, and an easy fix!

4 hours ago, Missy Charm said:

I wish to make a complaint.  Your supposed 'scraper', whatever that may be, is inherently sexist and, as a result, needs to be re-educated or terminated.  I attempted to look for a car under £2,000 in my area and the only result was this stereotypical girls' thing.  Keen observers will also note that the car is broken, presumably because the 'scraper' believes women shouldn't drive at all so is attempting to keep them off the road by displaying adverts only for inoperative vehicles.  This was not, and I repeat not, my fault for accidentally selecting 'Abarth' from the drop down list whilst it was loading!

This one is a less easy fix 😂

[Dan29]

This is brilliant!

I'm guessing you can't have an Any option in the Make and Model dropdowns, as it would return too many results?

And that there's a reason Autotrader returns a maximum of 10 results?

Excellent work.

[Merryck]

What have you built the site on? Happy to lend a hand with making it responsive etc. 

What format are you pulling the data through in? One thing I'd suggest (if it can handle it) would be allowing users to select multiple makes/models.

Also a sign up/login to allow users to save previous searches or set up alerts would be good.

[Dan29]

Also would it be possible to specify fuel and body type? And/or a keyword search? Please 🙂

[loserone]

1 hour ago, Dan29 said:

I'm guessing you can't have an Any option in the Make and Model dropdowns, as it would return too many results?

Not if combined with a 50 mile radius of the middle of nowhere and under a grand.  

 

How about browser notifications?

[moosemansjukebox]

1 hour ago, Dan29 said:

This is brilliant!

I'm guessing you can't have an Any option in the Make and Model dropdowns, as it would return too many results?

And that there's a reason Autotrader returns a maximum of 10 results?

Excellent work.

Thanks!

There's no technical reason it can't do any for make/model, I think this might be the next thing I have a stab at! 

There is a reason Autotrader only returns 10 results - it's because that's how many results you get per page on Autotrader I think it's possible to have it scrape more than one page, I just need to think about how to display it without it becoming a bit unwiedly. It'd be nice to blend and de-duplicate the results across all sites, but i'm not 100% sure how that would work just yet!
 

26 minutes ago, Merryck said:

What have you built the site on? Happy to lend a hand with making it responsive etc. 

What format are you pulling the data through in? One thing I'd suggest (if it can handle it) would be allowing users to select multiple makes/models.

Also a sign up/login to allow users to save previous searches or set up alerts would be good.

It's a Python Flask application, running on some self managed cloud infra. A hand making it more responsive would be appreciated! If you have any good tutorials/resources to point me at that'd be great, or i'm happy to share the Github repo with you if you like! 

The data is either scraped HTML or an API call, either way it results in a JSON object containing the common attributes. I think multiple makes/models might be possible - i'm effectively at the mercy of the sites I search, if they support it, I can support it - but they all have to support it. 

24 minutes ago, Dan29 said:

Also would it be possible to specify fuel and body type? And/or a keyword search? Please 🙂

I'll have a go at this with version 2 of the search filters!

[bangernomics]

You seem to have missed Opel off the list old chap.

Bloody great idea btw.


Could it get any worse?

[gadgetgricey]

Really good.

I think the Yanks already have something similar. "Autotempest.com"

Used it a few times when dreaming of buying an old Buick or Transam.

[cort16]

I've done  a version of this but I did it in a different way. I'm pretty sure to do it as a legit website you need to have agreements with the web site providers, which is why I only use mine for my own shite hunting purposes.

[Flat4]

Think I found a bit of a bug. Searching for an Alfa 156 or 159 less than 2 grand brings up 0 results, which I know is wrong coz there is at least 1 on Gumtree atm. I'm wondering if it's because you've got the word Alfa in the model, rather than just the number?

Bloody great idea tho! Thank you for making it

[LessThanEqual]

I had this exact same idea a couple of months ago but never wrote a line of code because of the duplication issue. I thought of maybe comparing the first thumbnail up to a certain confidence threshold or something like that. Gr9 work.

[808 Estate]

Brilliant. It did crash on me once and I found the "back to search" button a bit flaky a couple of times.

[Merryck]

I can't get on there at work, but if you send over the repo I'll have a look and see if I can tart it up a bit.

If you're getting the JSON through it might be simple to add a fuzzy search onto the search results page so users can search through the results they've pulled through for certain keywords and filter on that.

If the structure of the data is the same across all the sites then it should be possible to create a set of filters based on the information being pulled through (fuel type etc.) I'm dealing with a JSON-fed front end at the moment at work so if I come across anything helpful I'll be sure to pass it along!

[w00dy]

Great site, thanks for building it.

 

To get BMW models to come up in the search it seems to require "5 series" etc. in the title rather than the more common 530i or whatever.

[DVee8]

Thanks for this, I had a look around for a Suzuki Swift and a Renault Twingo. 

Can FB market place be added or is that to come?

[moosemansjukebox]

22 hours ago, cort16 said:

I've done  a version of this but I did it in a different way. I'm pretty sure to do it as a legit website you need to have agreements with the web site providers, which is why I only use mine for my own shite hunting purposes.

Yeah - I suspect you're right about the agreement stuff, and I suspect it's possibly why there's nothing out there already? Who knows - if i'm still interested in this project in a few months I might reach out to them and see what they say - but if I vanish from the internet entirely, I think it's safe to assume i've been assassinated by the used-car industry's hired goons
 

19 hours ago, Flat4 said:

Think I found a bit of a bug. Searching for an Alfa 156 or 159 less than 2 grand brings up 0 results, which I know is wrong coz there is at least 1 on Gumtree atm. I'm wondering if it's because you've got the word Alfa in the model, rather than just the number?

Bloody great idea tho! Thank you for making it

 

4 hours ago, w00dy said:

Great site, thanks for building it.

 

To get BMW models to come up in the search it seems to require "5 series" etc. in the title rather than the more common 530i or whatever.

The make/model list definitely needs some curation it seems! 

 

3 hours ago, DVee8 said:

Thanks for this, I had a look around for a Suzuki Swift and a Renault Twingo. 

Can FB market place be added or is that to come?

FB marketplace is an interesting one - i've only briefly looked, but they intentionally make it difficult for web scrapers to scrape the listings. And I totally understand why - it would circumvent their sponsored ads and tracking... 

[Dan_ZTT]

On 5/22/2022 at 9:31 PM, gadgetgricey said:

Really good.

I think the Yanks already have something similar. "Autotempest.com"

Used it a few times when dreaming of buying an old Buick or Transam.

Yeah AutoTempest is the US version of this idea, I only know it because they sponsor a bunch of youtubers who mention it constantly…

I doubt it’s something you can patent though so if you can make a decent UK version it could do very well indeed. Nothing to stop copycats though, ideally you’d build something functional quickly, get a bit of buzz going and then sell it for £££