NRD 2
      Back to home
      1. Knowledge Base
      2. NRD 2

      How do I get the list of newly registered domains on Linux/UNIX?

      How do I get a list of newly registered (dropped, discovered, etc.) domains from a CSV file in the Newly Registered Domains data feed (on Linux/UNIX)?

      Assume you are using the downloaded file "nrd.2022-07-23.ultimate.daily.data.csv.gz", and you want to get the domain names only in a text file.  To do so:

      1. Verify if the 'csvcut' utility is available on your system. If it isn't, install 'csvkit' with your package manager or Python 'pip'.
      2. Execute the command:
      zcat nrd.2022-07-20.ultimate.daily.data.csv.gz | csvgrep -c reason -m added | csvcut "domainName" > nrd.2022-07-20.ultimate.daily.domains.added.csv

      which will generate a csv with the domain names only.

       

      Remarks:

      • The traditional 'cut' command or 'awk' is not the best choice as the CSV files may contain quoted multiline fields.
      • Replace "added" with "dropped", "changed" or "discovered" to get the respective lists.
      • By simple modifications of the command line you can do more complex queries. For instance, to get both the newly registered and discovered domains and include the WHOIS registration date, you can execute the below:
      zcat nrd.2022-07-20.ultimate.daily.data.csv.gz | csvgrep -c reason -r '(added|discovered)' | csvcut -c "domainName,createdDateParsed" > nrd.2022-07-20.ultimate.daily.domains_with_regdate.added_and_discovered.csv
      • Consult the manals of `csvgrep' and `csvcut', and the specs of the Newly Registered Domains feed for additional ideas.