« first day (457 days earlier)      last day (2591 days later) » 

 
2 hours later…
 
1 hour later…
5:00 AM
@Queen k
 
 
4 hours later…
8:35 AM
Hi all .... I've an off topic question ....
I need to put one million files in 50,000 subfolders of a filesystem:
in your opinion what would be the best filesystem type for this kind of job ?
 
all in the same folder? large or small files? not really my field to be honest
if they are evenly distributed, 20 files per folder should be fine in any decent FS
if you need all 50,000 folders in one parent that could be a problem
I hear btrfs is pretty good with that but I have no direct experience
and AFAIK that's Linux only; you have said nothing about your platform
the usual workaround which should be fine for any filesystem is to add a hash fanout
so longfoldername and anotherlongfoldername go into lo/long/longfoldername and an/anot/anotherlongfoldername respectively, so you don't have many directories with many subdirectories, which is problematic in many traditional filesystems
and similarly for the files if you have many files in some of the folders
but thousands should be fine, tens of thousands probably still fine
 
9:02 AM
in 50,000 subfolders
they sum up to about 360Gb
on avarage they are going to be about 360kb
linux is fine
I did a bit of research and ext4 stands up fairly good on brtfs ... with the plus that it's very well tested and stable on linux
so I'm tempted to stick with ext4
 
sounds like a plan
maybe depends on your access patterns, too, do you want random access to every file all the time in parallel or do you process them sequentially one after another, etc
multiple physical drives could help performance if you need a lot of parallel I/O
 
9:27 AM
I don't have choice over the phys storage (it's on a SAN so it's not going to be crappy)
it's doing to be used for documentary portal (something in the lines of alfresco) ... I've no idea how much sequential and how much random access ... I would guess a good deal of both.
Possible a bit more sequential writing and random reading. But maybe at times when reporting is done there might be a good deal more sequential reading
 
actually might be a good question for serverfault.com if you can refine the spec a little bit more
 
10:01 AM
it would be a duplicate .... I've seen similar questions
I got inspiration from some of them
 
sure, but if it's specific enough and you've done your homework, you could get nice specific answers for your particulars
 
25
Q: What is the most high-performance Linux filesystem for storing a lot of small files (HDD, not SSD)?

user14159I have a directory tree that contains many small files, and a small number of larger files. The average size of a file is about 1 kilobyte. There are 210158 files and directories in the tree (this number was obtained by running find | wc -l). A small percentage of files gets added/deleted/rewrit...

 
the first comment is usually "did you measure and what did your measurements tell you?"
your files aren't small so that changes the game
1 message moved to friendly bin
 
I think I pasted wrong
I found hits in stackexchange, serverfaulr and stackoverflow
the goole search string I used is "linux best filesystem for many files and subfolders"
I generally make large filesystems with 4K blocks and 1 inode every 4 blocks ... but for 360kb average flie size I think I'd be safe to have an inode every 160Kb
360Kb may not be small but it's not big ither
I'd call it a silly size :D
do you agree that one inode every 160k is fine for an avarage size of 360k files ?
 
files which are smaller than your block size or just a little bit larger so they occupy two blocks with the second one nearly empty is something many people want to optimize. Compared to the file size the overhead of an incompletely populated block is negligible in your scenario
 
10:13 AM
on avarage you waste 1/2 a fs block for every file on the filesystem .... that's clear to me
but 4k is much smaller then the avarage file size so I'm not going to use smaller blocks
 
@louigi600 well yea, but when your files are small you leave almost half of your disk unused if the block size is a bad fit
@louigi600 don't really have the experience to support or refute your numbers
 
I'm more scheprical on the inodes .... on 1Tb i inoce every 160k should work out to be 6 and 1/4 million inodes ... which is only 6 times the amount of files I''m going to copy in there to start with
 
depends on the expected burn rate too, do you expect to create 6x that amount of files and never delete anything, or will there be turnaround, in which case 6x could be more than plenty
 
I should ask the middleware people what their intentions are :D
I told them that 1M files is a 10 year history on the previous documentary .... and then asked them if 6M is enough inodes for the new one .... but I'm not sure they even know what an inode is :D
 
disk is cheap and with today's hardware migrating 1Tb from one disk to another if you really need to isn't altogether a major deal
60 years sounds like you should have margins to spare
 
10:32 AM
I think it's should be fair enough ... but you never know if the new application is going to make lots of smaller files. I'll wait for some sort of answer from the middlware people.
friendly bin ?
i redid my maths with more accurate numbers:
1099511627776 / 163840
6710886.40000000000000000000
that's closer to 7M inodes on the filesystem
 
10:47 AM
@louigi600 room owners can relocate messages but not delete them ... the "trashbin" name they used to have for the garbage room apparently got some people upset, so they have a less trashy name for it now
 
11:25 AM
I'm not complaining .... I was asking if this was good fro the friendly bin ?
I'm well ware that this is off toppic on bash chat ... and I'm thankfull that you have participated anyway with usefull suggestions ;)
 
12:25 PM
no prob
 
@Queen k
 
 
7 hours later…
7:54 PM
Ugh, can we nuke this one?
0
Q: Ubuntu Linux and Terminal tutorials

Bejan Muhidinovi'm a newbee in Ubuntu Linux and i've changed Windows 10 to Ubuntu Linux. Now my question is: How do i learn to work with Linux ? I have seen some tutorials on web, but most of the time they're useless. Could you please suggest any good tutorial. And i know that many tasks in linux are done by ...

 

« first day (457 days earlier)      last day (2591 days later) »