Watch Entire Directory with NIO WatchService

Discussion in 'It's My Way' started by dangxunb, 15/2/16.

  1. dangxunb

    dangxunb Active Member

    I've created a WatchDirService class to simplicity the effort with NIO's WatchService. I also added the feature to watch entire classes include subdir. The process of recursively register directory with WatchService is a lightweight process and somewhat interesting me, so I include it into my class.

    Usage Example:
    PHP:
    import java.io.IOException;
    import java.nio.file.Path;
    import java.nio.file.Paths;
    import java.nio.file.StandardWatchEventKinds;


    /**
    *
    * @author NguyenHaiDang
    */
    public class Test {
        public static 
    void main(String[] argsthrows IOExceptionInterruptedException {
            
    //monitor CREATE and DELETE event
            
    WatchDirService watchService = new WatchDirService(trueStandardWatchEventKinds.ENTRY_CREATE,
                                                                
    StandardWatchEventKinds.ENTRY_DELETE);
       
            
    //register a directory, can call multiple times to register multiple directories
            
    watchService.registerDir(Paths.get("E:\\"));
       
            
    //register an action when a new file/dir created
            
    watchService.setOnCreatedAction(new WatchEventAction() {
                @
    Override
                
    public void doAction() {
                    
    //get the Path object of the file which make event happent
                    
    Path path watchService.getCurrentFilePath();
               
                    
    //print out its absolute path
                    
    System.out.println(path.toAbsolutePath().toString());
                }
            });
       
            
    //register an action when a new file/dir deleted
            
    watchService.setOnDeletedAction(new WatchEventAction() {
                @
    Override
                
    public void doAction() {
                    
    //get the Path object of the file which make event happent
                    
    Path path watchService.getCurrentFilePath();
               
                    
    //print out its absolute path
                    
    System.out.println(path.toAbsolutePath().toString());
                }
            });
       
            
    //start it in a seperate thread
            
    new Thread(watchService).start();
        }
    }

    Source code: Gist
    PHP:
    package com.activestudy.Utility.file.watchservice;

    import java.io.IOException;
    import java.nio.file.*;
    import java.util.*;
    import static java.nio.file.StandardWatchEventKinds.*;
    import java.nio.file.WatchEvent.Kind;
    import java.nio.file.attribute.BasicFileAttributes;

    /**
     * A class based on NIO's WatchService to watch entire folder.
     *
     * @author NguyenHaiDang@ActiveStudy
     */
    public class WatchDirService implements Runnable {

        private final WatchService watchService;
        private final Map<WatchKey, Path> keyPathMap = new HashMap<>();
        private final WatchEvent.Kind[] eventKinds;
        private Path currentFilePath;
        private WatchEventAction onCreatedAction;
        private WatchEventAction onModifiedAction;
        private WatchEventAction onDeletedAction;
        private final boolean all;

        public WatchDirService(boolean all, WatchEvent.Kind... eventKinds) {
            try {
                this.watchService = FileSystems.getDefault().newWatchService();
            } catch (IOException ex) {
                throw new RuntimeException("Cannot create watch service. Reason: ", ex);
            }
            this.eventKinds = eventKinds;
            this.all = all;
        }

        /**
         * This blocking method continously watch given directories until an
         * exception throw or it have no job to do anymore (ex: folder deleted,
         * permission changed...)
         *
         * @throws IOException
         * @throws InterruptedException
         */
        private void watch() throws IOException, InterruptedException {
            MAIN_LOOP:
            while (true) {
                final WatchKey key = watchService.take();

                INNER_LOOP:
                for (WatchEvent<?watchEvent key.pollEvents()) {
                    final 
    Kind<?> kind = watchEvent.kind();

                    WatchEvent<Path> watchEventPath = (WatchEvent<Path>) watchEvent;
                    Path fileName = watchEventPath.context();
                    Path parentDir = keyPathMap.get(key);
                    Path child = parentDir.resolve(fileName);
                    setCurrentFilePath(child);

                    if (kind == ENTRY_CREATE) {
                        if (Files.isDirectory(child, LinkOption.NOFOLLOW_LINKS) && all) {
                            registerDir(child);
                        }
                        performAction(onCreatedAction);
                        continue;
                    }

                    if (kind == ENTRY_MODIFY) {
                        performAction(onModifiedAction);
                        continue;
                    }

                    if (kind == ENTRY_DELETE) {
                        performAction(onDeletedAction);
                    }
                }

                boolean validKey = key.reset();

                if (!validKey) {
                    keyPathMap.remove(key);
                    if (keyPathMap.isEmpty()) {
                        watchService.close();
                        break MAIN_LOOP;
                    }
                }
            }
        }

        private void registerPath(Path path) throws IOException {
            WatchKey key = path.register(watchService, eventKinds);
            keyPathMap.put(key, path);
        }

        /**
         * Register a directory to watch
         *
         * @param dir Path, the Path object represent the directory
         * @throws IOException
         */
        public void registerDir(Path dir) throws IOException {
            if (Files.isDirectory(dir, LinkOption.NOFOLLOW_LINKS)) {
                if (all) {
                    Files.walkFileTree(dir, new SimpleFileVisitor<Path>() {
                        @Override
                        public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs)
                                throws IOException {
                            registerPath(dir);
                            return FileVisitResult.CONTINUE;
                        }

                        public FileVisitResult visitFileFailed(Path file, IOException exc)
                                throws IOException {
                            return FileVisitResult.CONTINUE;
                        }
                    });
                } else registerPath(dir);
            }
        }

        private void setCurrentFilePath(Path file) {
            currentFilePath = file;
        }

        /**
         *
         * @return the Path object of the file which is making current event
         */
        public Path getCurrentFilePath() {
            return currentFilePath;
        }

        public void setOnCreatedAction(WatchEventAction onCreatedAction) {
            this.onCreatedAction = onCreatedAction;
        }

        public void setOnModifiedAction(WatchEventAction onModifiedAction) {
            this.onModifiedAction = onModifiedAction;
        }

        public void setOnDeletedAction(WatchEventAction onDeletedAction) {
            this.onDeletedAction = onDeletedAction;
        }

        private void performAction(WatchEventAction ea) {
            ea.doAction();
        }

        @Override
        public void run() {
            try {
                watch();
            } catch (IOException | InterruptedException ex) {
                throw new RuntimeException(ex);
            }
        }
    }
    The WatchEventAction interface:
    PHP:
    /**
    * Provide an action when a WatchEvent happen
    * @author NguyenHaiDang
    */
    public interface WatchEventAction {
        
    void doAction();
    }
    I don't see any sensible performance problem with this API. As the example above, I register entire E:\ drive with it (79.5GB size, ~2500 file, 1100 folder) and quickly making event in a deep directory. It interact very fast. No overhead at start time.
    I appropriate any help to improve this class design and effeciency. If forum's admins accept this, I hope it can join congdongjava's library. Thank you :D
  2. Joe

    Joe Moderator

    Good work, but it's the Job of JackV or Tranhuyvc. I'm just a guest admin with limited admin-rights. That's good so and suits me.
  3. dangxunb

    dangxunb Active Member

    My main purpose when I upload this is to learn class design and code effeciency :D Add to congdongjava library to help people who want to easily use it is admin's job :)
  4. Joe

    Joe Moderator

    You don't seem to read my comment. I said "limited admin-rights" and that meant a lot.
  5. dangxunb

    dangxunb Active Member

    Joe, I know that, and I don't mean anything, esspecially I don't point to you :( Again, I'm happy here to learn and share my knowledge, mostly from you, and don't really care much about other things :(
  6. Nancru

    Nancru CongDongJava Project Leader Staff Member

    The performance is good because your IO is low, how about you create an application to read multiple files at the same time(let's say 10 files a time, start with small numbers or your PC will be froze). And see what happens?

    On Windows, to monitor IO you can type perfmon in cmd
    On Linux, you can type iostats -x 1 in terminal to check it
  7. dangxunb

    dangxunb Active Member

    I think you should read carefully the WatchService API and you will understand why it's fast.
    My API is just based on the WatchService API and I was said that
    With the test above, when the app first run, IO jump slowly from 0% to 18% in 2.5s (because of the recursive register dir) then the disk IO come back to 0% when it's monitoring. I think it's an acceptable performance with entire drive E:\. I posted it here to improve it. Please share with me. Thank you :)
  8. Nancru

    Nancru CongDongJava Project Leader Staff Member

    Lol, Sorry, I mean your IO is low, not slow. I mean that you should test when your PC's IO is high or very high, or at 100%. The best way to test is Let It Crash.
    I sometimes face with the IO problems, one time I feel hopeless when all the optimization I gave in my codes, but the system is still crash.
    dangxunb likes this.
  9. dangxunb

    dangxunb Active Member

    Thank you :D Can you give me some simple way to make my PC IO work at 100%? :D
  10. Nancru

    Nancru CongDongJava Project Leader Staff Member

    Read some files at the same time, you can make an application to read files with multiple threads.
    And make endless loop in every thread to keep it always busy.
    However I hope that your CPUs are powerful, otherwise it's hard to test it when your number of CPUs is just about two and they are weak. In that case, you are testing the CPU performance.
  11. Joe

    Joe Moderator

    Hi all,
    The recursive algorithm is RARELY the best way to obtain the best performance, it's merely the best way to solve a problem. That's the principle difference and it's one of the basic POPL -Problem Oriented Programming Language. The best method to solve trickiest problem is rarely be the same to the best way to obtain the most optimal performance. IOs are always the same -regardless of the way how they are read. The reason is simple: Modern CPU processing time is million times faster than one IO access.

    @Dangxunb
    As I said, only the bosses, JackV and Tranhuyvc, can decide what member's software is "worth" for CongdongJava Library. I abstain from such a decision. OK?

Chia sẻ trang này