#python: list de-duping, list of lists batches
▻https://hackernoon.com/python-list-de-duping-list-of-lists-batches-75ffa389cd7?source=rss----3a
Lists list list I love lists, just about everything can be a list of something. There are a thousand ways to solve problems with lists. Let’s talk about de-duping a long list and then cutting up that list into batches.The problem encountered was a list of some 60,000 IDs and in that list there were 3 dupes. Then after getting a clean list, that list needed to be batched up with 5000 IDs per batch.The initial idea was to build iterating loops to cycle through the list to find the dupes, using a recursive loop in a loop. This was going to be tedious and perhaps 15 lines of code, pretty standard stuff. To the google, and of course, ended up on stack overflow. There is a python functionality called “set”.8.7. sets - Unordered collections of unique elements - Python 2.7.15 (...)