Create ZIP of CSV files

Category: c# general

Question

Dotnetavalanche on Fri, 29 Jun 2018 16:36:47


I am working in .Net 4.7.2. We have List of objects say MyObject which is to be converted to csv single file. Currently using below code I create HUGE csv file ( 10GB and onwards).

using (var writ = new StreamWriter(fileStream, Encoding.UTF8)) { using (var csvWrit = new CsvWriter(writ)) { //logic to read one object

// write to csv

csvWrit.NextRecord(); } } ZipFile.CreateFromDirectory(<sourcefileName>, <destFileName>);

Now i need to create zip of these HUGE file. I found ZipFile.CreateFromDirectory in C#. After csv is created i call ZipFile.CreateFromDirectory to create zip file.

My Question Should i continue to first create the csv and then zip OR Any other efficient way to do this?



Replies

CoolDadTx on Fri, 29 Jun 2018 17:43:16


.zip files are just compressed files so if you want a .zip file you'd need the csv file first. However if you just need to compress data and you don't care about it being extractable to a csv then you can also consider using one of the compression stream classes instead. It would allow you to stream the data directly to a compressed file. DeflateStream is based upon zlib and is pretty fast. Note that these types of classes aren't designed to produce files you can then extract files from, they are designed to compress data so the resulting file would be smaller but you wouldn't just be able to open the file and access the underlying data.

Viorel_ on Fri, 29 Jun 2018 20:29:56


The next fragment seems to work:

using( var zf = ZipFile.Open( "myfile.zip", ZipArchiveMode.Create ) )
{
   var ze = zf.CreateEntry( "myfile.csv" );
   using( var zs = ze.Open() )
   {
      using( var writ = new StreamWriter( zs, Encoding.UTF8 ) )
      {
         // write to 'writ'
         // . . .
      }
   }
}


kethineni.vamsi on Tue, 05 Nov 2019 05:49:13


Hi  Dotnetavalanche,

I Have a requirement to export data as csv, zip it and give it for download. So, considering the huge data around 100,000+ records, is it best to use the above code or do we need to choose a different path? please suggest.

Thanks,

Vamsi K