java - Proper way to handle file writes in separate threads -

i looking write files multiple times (100k+) , writes happen on flaky network. this, considering using java executorservice generate threads, i'm not quite sure combination of settings make following happen:

1. only allow 1 write happen @ time (order matters of course)
2. allow write ample time conduct each write (say 5 second) @ point bail
3. if writing slow, have executor collect writes in queue , wait.
4. don't allow overall program exit until thread queue empty.
5. separate threads writers. i.e., if same exact writer comes in function, put in own queue. if different writer pointer comes in, give own queue (no need put separate writers in same queue).

i believe can done combination of executor features along .wait() , .notify() command on main program's object. however, not sure how precisely work executor api done.

here got:

private void writetofileinseperatethread(final printwriter writer, final string text) {   executorservice executor = executors.newsinglethreadexecutor();   try {     executor.submit(new thread(new runnable() {       public void run() {         writer.println(text);       }     })).get(5l, timeunit.seconds);   } catch (exception e) {     e.printstacktrace();   }   executor.shutdown(); } 

that method called 100k+ times during single process, i'm not sure if should creating new excutorservice instance each time, or utilizing same one? (in attempts utilize same one, kept getting exceptions believe related .newsinglethreadexecutor() directive.

would stay java 5 compliant java 6 okay. running on windows xp/7.

update: seems have done trick in initial testing:

  private class writerstringpair {     public final printwriter writer;     public final string text;      public writerstringpair(printwriter writer, string text) {       this.writer = writer;       this.text = text;     }   }    private void writetextinseperatethread(writer writer, string text) {     try {       textqueue.offer(new writerstringpair(writer, text), 300l, timeunit.seconds);     } catch (interruptedexception e) {       errout.println(e);       e.printstacktrace();     }   }    final blockingqueue<writerstringpair> textqueue = new arrayblockingqueue<writerstringpair>(500);    private void setwritingthread() {     new thread((new runnable() {       public void run() {         writerstringpair q;         while (!shutdown && !thread.currentthread().isinterrupted()) {           try {             q = textqueue.poll(1l, timeunit.seconds);             if (q != null) {               q.writer.write(q.text + "\n");               q.writer.flush();             }           } catch (exception e) {             e.printstacktrace();           }         }       }     })).start();   } 

without knowing more details writing files on "flaky" network , means, it's hard give specifics. here things think about.

i figure out number of concurrent writers gives best performance here -- or reliable output on destination. should start fixed number of these writers, each consuming shared blockingqueue (or 1 queue per writer if matters). should exceed io or network bandwidth starting @ 5 or writers , going or down necessary should work.

public void run() {    writer.println(text); } 

yeah, don't want sort of thing in terms of job per line. better put string text blockingqueue<string> , have writer runnable classes running in executorservice dequeueing queue , stopping when queue empty or shutdown boolean set.

as peter mentions, need careful filling memory queued text strings. if input text large, should set limit on blockingqueue couple of hundred or so.

i'm not sure if should creating new executorservice instance each time, or utilizing same one?

certainly should have single service , not create 1 on , over.

i believe can done combination of executor features along .wait() , .notify() command on main program's object.

you shouldn't need use wait , notify if write correctly. i'd have volatile boolean shutdown = false of writers watch. each of them dequeues text queue using looking @ shutdown. like:

while (!shutdown && !thread.currentthread().isinterrupgted()) {     string text = textqueue.poll(1, timeunit.seconds);     if (text != null) {         // write text     } } 

if write fail or can retry or whatever necessary.