Questions tagged [csv]

A comma-separated values (CSV) file stores tabular data (numbers and text) as plain text, with rows and columns delimited by line terminators and commas, respectively. You may also use this tag for variants where fields are separated by tabs or semicolons.

Filter by
Sorted by
Tagged with
2
votes
1answer
654 views

Reading multiple CSV files and asynchronously update SQLite

My console app needs to read several big CSV files to update its data, which are stored in a SQLite database. I am new to C# and would like to know if there are ways to improve this sample code I ...
0
votes
2answers
2k views

Best way to get the best performance of a large CSV upload in PHP

I am trying to upload a CSV file to a database which has around 800,000 rows. I am able to do this on my server with the following specs: RAM: 4GB Rows: 800,000 Time taken: 305 seconds Usage: 15% RAM ...
5
votes
1answer
335 views

Read CSV into 2D float array in Go

Depending on how you count, this is my first Go program. I'm trying to read a CSV into a two-dimensional array of some numeric type, and then print it out. (I want to use this to read "edge weights" ...
3
votes
2answers
2k views

Delete lines from a CSV file that contain fields listed in a text file

I wrote a powershell script to compare words from a text-file with a csv-column. If the word in the column matches, the line is deleted. ...
6
votes
1answer
96 views

Counting weather events for visualization using Gnuplot

I wrote this code to analyze ~800 MB of weather data from the US. I am planning on visualizing the data with Gnuplot and Gimp. I have already made the images and a gif file. This code runs somewhat ...
3
votes
3answers
4k views

Displaying a leader board from a text file and calculating points

The text file is laid out like this: first name, last name, wins, losses dex,kop,5,6 desder,jok,0,11 aslan,vert,11,0 jax,sas,2,9 buy,guy,1,10 Code: ...
5
votes
1answer
141 views

Find the maximum line length in a given TSV colum

I'm trying to write a function that given a TSV file will get the maximum line length found in a column. I currently have this (working) function. However, I feel that I am missing something as this ...
5
votes
5answers
28k views

Simple class to create csv

As part of a solution I have the following class createCSV. It is working as expected but I'm very much aware of my lack of experience with ...
4
votes
1answer
3k views

Login program in Python

I wrote this login program as part of my project for school. Should I use my implementation of a login program, or is there some standardized patterns of login programs, like the ones in encryption ...
6
votes
2answers
1k views

Reading large amount of player properties from CSV

I have written a class with a function that reads out the CSV from the webpage and creates an object with it. The class seems too big - it seems very unoptimized, prone to mistakes and it's not very ...
3
votes
1answer
1k views

Calculate stock portfolio profit based on CSV data

import time import csv import argparse from nsetools import Nse from colorama import Fore nse = Nse() t = time.time() CSV file contains the following data, I ...
5
votes
1answer
1k views

Simple query language for CSV files

I'm struggling to improve this command-line java application, which reads CSV or txt files, ands allows the user do some "queries" with the data. I can't use any external library, so I'm doing it all ...
7
votes
1answer
2k views

Efficiently adding huge amounts of data from CSV files into an SQLite DB in Java

I'm trying to parse values from a CSV file to a SQLite DB, however the file is quite large (~2,500,000 lines). I ran my program for a few hours, printing where it was up to, but by my calculation, the ...
0
votes
1answer
4k views

Searching a CSV with PHP

Given a string, I've done the below to search a CSV for a keyword and display the row if it's found ...
4
votes
3answers
14k views

Finding the states with the three most populous counties

I just started to use Python and Pandas. My current solution to a problem looks ugly and inefficient. I would like to know how to improve it. Data file is Census 2010 can be viewed here Question: ...
5
votes
1answer
70 views

Plotting some displays from a weather URL

I've got some code that plots some displays from a weather URL and locations in a CSV file. I'd like to see if anyone can make the code more efficient, the code runs fine without any errors and I'm ...
11
votes
4answers
2k views

Country code lookup for each line in a CSV file

I'm trying to write a program that has a for loop, inside of which I have lambda expression to get a certain value. The loop has about 20,000 iterations. Including ...
5
votes
2answers
2k views

java class to read csv files

Here is my class to read a csv file and I was wondering if there is any improvements that I should do. I feel there are some performance improvements or in exception handling because I just took IO ...
4
votes
1answer
14k views

Writing CSV file from huge JSON data

I am writing a program that reads from DB and outputs to a CSV file. Besides the regular columnar data there are 2 JSON fields data as well. The table layout looks like this (other fields removed for ...
4
votes
0answers
268 views

PowerShell Export-CSV with Cultural Awareness

I recently had a requirement to export data to CSV in a different culture & hit an issue (see https://stackoverflow.com/a/40871162/361842). Here's my workaround: ...
5
votes
1answer
191 views

Frequency analysis for counting keywords

I have a CSV with around 10,000,000 lines. Each line looks like: ...
2
votes
1answer
263 views

Multiple files data processing in Clojure

I have made a small script which grabs data into a map from 80 CSV files and calculates some statistics like average, standard deviation etc. It's also adding some additional data to map from filename....
2
votes
3answers
5k views

Converting a text file to a CSV file

I'm attempting to learn more about Java and have created a method that takes a text file with stdout (space separated file) and converts it to a CSV file. I was ...
0
votes
1answer
8k views

Read a CSV file and do natural language processing on the data

I am studying the techniques of data mining and data processing. I'm doing this through data I've collected and stored in a csv file. The problem is that this filed was very large, to the point of ...
3
votes
1answer
73 views

ECG Bash selection tool

I made the following bash script for extracting a group of ECG signals from ECG files. I would like to know if there is any mistakes and/or weaknesses. I have experienced difficulties in integrating ...
2
votes
1answer
3k views

Performing a regex search and saving results to CSV

I am a Python beginner, and I wrote a simple Python program, which does the following: search the pattern in the lines of a file (contains messages) pick the information from the lines and save it to ...
3
votes
1answer
81 views

Gather information about computers from multiple CSV files

I created a script to import several CSV files from various sources and one CSV file with a list of systems in it. the script searches each CSV file to see if the system exist in the file and if it ...
5
votes
1answer
112 views

Calculating a table of deBroglie wavelengths for various electron energies

Here is the formula for the deBroglie wavelength of an electron versus its kinetic energy: $$ \lambda(E_k) = h\left/\sqrt{\frac{(E_k+m_eC^2)^2-m_e^2C^4}{C^2}}\right.$$ and here is simple script that ...
5
votes
1answer
555 views

Perl script to analyze CSV files

I made a Perl script to analyze CSV files. However, it is running a little slow. Is there anything that I can change to make it more efficient? ...
3
votes
2answers
223 views
2
votes
1answer
205 views

Arduino weather station logger

I made a program that uses the Adafruit pressure sensor, and the DHT22 temperature and humidity sensor to log pressure, temperature and humidity every 30 minutes. I left it outside my house overnight ...
3
votes
0answers
105 views

Fetching certain eventIDs from servers on network.

The PowerShell script below queries the Security event log on one or more servers for events with id 4663. When trying to retrieve all audit events for event id 4663 with the following code the ...
4
votes
2answers
234 views

Generate SQL UPDATE from Excel CSV file

I must write updates in a database from a flat file (CSV). I want to do that in the shell, with tools such as AWK. ...
6
votes
2answers
220 views

~600k external API requests for uploading part numbers to an inventory

I have a CSV containing ~600k part numbers to be uploaded to my website's inventory. However, this CSV only contains limited information. We're missing pricing and other related information. To get ...
3
votes
1answer
126 views

System to process data from a specific client

I'm working on an important project. We're an IT services provider and we have a project where we have to build a new system to process data from a specific client. The problem is that usually we ...
3
votes
1answer
7k views

Extracting specific rows and columns from a CSV file

I have written a function to selectively extract data from a file. I want to be able to extract only from a certain line and only given rows. Would convert this function into a generator improve the ...
1
vote
1answer
121 views

Streaming learning OCaml

I wrote a simple online logistic regression, calibrated using gradient descent to compare the speed of an OCaml implementation vs the same Python script, executed with Pypy. It turned out that the ...
3
votes
0answers
2k views

Pandas calculation speed of stock beta on many dataframes

I have many (4000+) CSVs of stock data (Date, Open, High, Low, Close) which I import into individual Pandas dataframes to perform analysis. I am new to Python and want to calculate a rolling 12month ...
10
votes
2answers
1k views

Reading a repetitive file with repetitive code

I made this code for take some data from a .txt file. The text file is a large list of data from temperature with a format that is repetitive and have some lines at the beginning with the station ...
3
votes
1answer
7k views

PowerShell script to read line by line large CSV files

I am managing large CSV files (files ranging from 750 Mb to 10+ Gb), parsing their data into PSObjects, then processing each of those objects based on what is required. I wrote the following script ...
1
vote
2answers
53 views

Generate Orders from Matrix and save as CSV

I had a task to create a VBA macros which will take a matrix data set with products in rows and persons as a columns (on intersection we have an ordered qty for each material for each person) and ...
3
votes
1answer
163 views

Imported data from CSV successfully, but slow

Currently I am using the gem activerecord-import to seed more than 55,000 records into my database, and it works fine. However, it takes too much time. I am ...
3
votes
2answers
59 views

CSV to Object converter with JavaDoc comments

This is a simple class but I'm looking to check I have the basics down to scratch. Is my documentation thorough enough? (I'm very new to writing JavaDoc) Have I created too many variables, and does ...
1
vote
2answers
111 views

Comparing adjacent time stamps

I have this code which compares two adjacent time stamps. If they are less than 15 min apart, it adds a new time stamp (row) in between with the new time stamp equal to the previous time + 15 min and ...
3
votes
1answer
10k views

Basic C++ CSV class to read and write to .csv files

I've been working through the fifth edition of Lajoie's C++ Primer and decided to write a CSV reader/writer as my first project. Its main features are: Able to explicitly call for: Individual/...
6
votes
3answers
236 views

CSV concatenator

I have the following code (running in LINQPad) which takes tens of thousands of CSV files and concatenates them to create a single file. Each CSV file has two lines: a header line and a data line. I ...
4
votes
1answer
1k views

CSV file parser and compare

This may seem like a lot of stuff? I just need help with 2 small parts the code works, however I have provided the rest of the info in case some one can help. USING PYTHON 3.4 Code below is ...
6
votes
2answers
818 views

Showing virus scan results from an API and a CSV file

I am utilizing an API module for interacting with virustotal.com in order to get AntiVirus results based of SHA256 hashes. The code I have is working but I feel like it can be improved greatly. I ...
5
votes
1answer
523 views

Powershell Interaction with Excel (Highlighting Cells)

I'm using Powershell to compare two, large csv files and output the rows which don't match. Once the two output files are made (one for the differences on each), I bring them up on Excel. Now I want ...
1
vote
2answers
2k views

PHP script that combines data from two CSV files to produce two new files

I've spent some time looking at various posts on Stack Overflow regarding PHP CSV file manipulation and have devised a script that combines data from two CSV files to create two new files. The aim of ...