Package org.apache.cassandra.db.compaction

Class CompactionManager

  • All Implemented Interfaces:
    CompactionManagerMBean
    public class CompactionManager
extends java.lang.Object
implements CompactionManagerMBean

    A singleton which manages a private executor of ongoing compactions.

    Scheduling for compaction is accomplished by swapping sstables to be compacted into a set via Tracker. New scheduling attempts will ignore currently compacting sstables.

    • Constructor Detail

      • CompactionManager

        public CompactionManager()

    • Method Detail

      • getRateLimiter

        public com.google.common.util.concurrent.RateLimiter getRateLimiter()
        Gets compaction rate limiter. Rate unit is bytes per sec.
        Returns:
        RateLimiter with rate limit set

      • setRate

        @Deprecated
public void setRate​(double throughputMbPerSec)
        Deprecated.
        Use setRateInBytes instead
        Sets the rate for the rate limiter. When compaction_throughput is 0 or node is bootstrapping, this sets the rate to Double.MAX_VALUE bytes per second.
        Parameters:
        throughputMbPerSec - throughput to set in MiB/s

      • setRateInBytes

        public void setRateInBytes​(double throughputBytesPerSec)
        Sets the rate for the rate limiter. When compaction_throughput is 0 or node is bootstrapping, this sets the rate to Double.MAX_VALUE bytes per second.
        Parameters:
        throughputBytesPerSec - throughput to set in B/s

      • submitBackground

        public java.util.List<Future<?>> submitBackground​(ColumnFamilyStore cfs)
        Call this whenever a compaction might be needed on the given columnfamily. It's okay to over-call (within reason) if a call is unnecessary, it will turn into a no-op in the bucketing/candidate-scan phase.

      • isCompacting

        public boolean isCompacting​(java.lang.Iterable<ColumnFamilyStore> cfses,
                            java.util.function.Predicate<SSTableReader> sstablePredicate)

      • hasOngoingOrPendingTasks

        public boolean hasOngoingOrPendingTasks()

      • forceShutdown

        public void forceShutdown()
        Shutdowns both compaction and validation executors, cancels running compaction / validation, and waits for tasks to complete if tasks were not cancelable.

      • finishCompactionsAndShutdown

        public void finishCompactionsAndShutdown​(long timeout,
                                         java.util.concurrent.TimeUnit unit)
                                  throws java.lang.InterruptedException
        Throws:
        java.lang.InterruptedException

      • getBackgroundCompactionCandidate

        public org.apache.cassandra.db.compaction.CompactionManager.BackgroundCompactionCandidate getBackgroundCompactionCandidate​(ColumnFamilyStore cfs)

      • performScrub

        public CompactionManager.AllSSTableOpStatus performScrub​(ColumnFamilyStore cfs,
                                                         boolean skipCorrupted,
                                                         boolean checkData,
                                                         int jobs)
                                                  throws java.lang.InterruptedException,
                                                         java.util.concurrent.ExecutionException
        Throws:
        java.lang.InterruptedException
        java.util.concurrent.ExecutionException

      • performScrub

        public CompactionManager.AllSSTableOpStatus performScrub​(ColumnFamilyStore cfs,
                                                         boolean skipCorrupted,
                                                         boolean checkData,
                                                         boolean reinsertOverflowedTTL,
                                                         int jobs)
                                                  throws java.lang.InterruptedException,
                                                         java.util.concurrent.ExecutionException
        Throws:
        java.lang.InterruptedException
        java.util.concurrent.ExecutionException

      • performSSTableRewrite

        public CompactionManager.AllSSTableOpStatus performSSTableRewrite​(ColumnFamilyStore cfs,
                                                                  boolean skipIfCurrentVersion,
                                                                  long skipIfOlderThanTimestamp,
                                                                  boolean skipIfCompressionMatches,
                                                                  int jobs)
                                                           throws java.lang.InterruptedException,
                                                                  java.util.concurrent.ExecutionException
        Throws:
        java.lang.InterruptedException
        java.util.concurrent.ExecutionException

      • performSSTableRewrite

        public CompactionManager.AllSSTableOpStatus performSSTableRewrite​(ColumnFamilyStore cfs,
                                                                  java.util.function.Predicate<SSTableReader> sstableFilter,
                                                                  int jobs)
                                                           throws java.lang.InterruptedException,
                                                                  java.util.concurrent.ExecutionException
        Perform SSTable rewrite
        Parameters:
        sstableFilter - sstables for which predicate returns {@link false} will be excluded
        Throws:
        java.lang.InterruptedException
        java.util.concurrent.ExecutionException

      • performCleanup

        public CompactionManager.AllSSTableOpStatus performCleanup​(ColumnFamilyStore cfStore,
                                                           int jobs)
                                                    throws java.lang.InterruptedException,
                                                           java.util.concurrent.ExecutionException
        Throws:
        java.lang.InterruptedException
        java.util.concurrent.ExecutionException

      • relocateSSTables

        public CompactionManager.AllSSTableOpStatus relocateSSTables​(ColumnFamilyStore cfs,
                                                             int jobs)
                                                      throws java.util.concurrent.ExecutionException,
                                                             java.lang.InterruptedException
        Throws:
        java.util.concurrent.ExecutionException
        java.lang.InterruptedException

      • performAnticompaction

        public void performAnticompaction​(ColumnFamilyStore cfs,
                                  RangesAtEndpoint replicas,
                                  Refs<SSTableReader> validatedForRepair,
                                  LifecycleTransaction txn,
                                  TimeUUID sessionID,
                                  java.util.function.BooleanSupplier isCancelled)
                           throws java.io.IOException
        Make sure the {validatedForRepair} are marked for compaction before calling this. Caller must reference the validatedForRepair sstables (via ParentRepairSession.getActiveRepairedSSTableRefs(..)).
        Parameters:
        cfs -
        replicas - token ranges to be repaired
        validatedForRepair - SSTables containing the repaired ranges. Should be referenced before passing them.
        sessionID - the repair session we're anti-compacting for
        isCancelled - function that indicates if active anti-compaction should be canceled
        Throws:
        java.lang.InterruptedException
        java.io.IOException

      • performMaximal

        public void performMaximal​(ColumnFamilyStore cfStore,
                           boolean splitOutput)

      • submitMaximal

        public java.util.List<Future<?>> submitMaximal​(ColumnFamilyStore cfStore,
                                               int gcBefore,
                                               boolean splitOutput)

      • forceCompaction

        public void forceCompaction​(ColumnFamilyStore cfStore,
                            java.util.function.Supplier<java.util.Collection<SSTableReader>> sstablesFn,
                            com.google.common.base.Predicate<SSTableReader> sstablesPredicate)

      • forceCompactionForTokenRange

        public void forceCompactionForTokenRange​(ColumnFamilyStore cfStore,
                                         java.util.Collection<Range<Token>> ranges)
        Forces a major compaction of specified token ranges of the specified column family.

        The token ranges will be interpreted as closed intervals to match the closed interval defined by the first and last keys of a sstable, even though the Range class is suppossed to be half-open by definition.

        Parameters:
        cfStore - The column family store to be compacted.
        ranges - The token ranges to be compacted, interpreted as closed intervals.

      • forceUserDefinedCompaction

        public void forceUserDefinedCompaction​(java.lang.String dataFiles)
        Description copied from interface: CompactionManagerMBean
        Triggers the compaction of user specified sstables. You can specify files from various keyspaces and columnfamilies. If you do so, user defined compaction is performed several times to the groups of files in the same keyspace/columnfamily.
        Specified by:
        forceUserDefinedCompaction in interface CompactionManagerMBean
        Parameters:
        dataFiles - a comma separated list of sstable file to compact. must contain keyspace and columnfamily name in path(for 2.1+) or file name itself.

      • forceUserDefinedCleanup

        public void forceUserDefinedCleanup​(java.lang.String dataFiles)
        Description copied from interface: CompactionManagerMBean
        Triggers the cleanup of user specified sstables. You can specify files from various keyspaces and columnfamilies. If you do so, cleanup is performed each file individually
        Specified by:
        forceUserDefinedCleanup in interface CompactionManagerMBean
        Parameters:
        dataFiles - a comma separated list of sstable file to cleanup. must contain keyspace and columnfamily name in path(for 2.1+) or file name itself.

      • submitValidation

        public Future<?> submitValidation​(java.util.concurrent.Callable<java.lang.Object> validation)

      • disableAutoCompaction

        public void disableAutoCompaction()

      • needsCleanup

        public static boolean needsCleanup​(SSTableReader sstable,
                                   java.util.Collection<Range<Token>> ownedRanges)
        Determines if a cleanup would actually remove any data in this SSTable based on a set of owned ranges.

      • submitIndexBuild

        public Future<?> submitIndexBuild​(SecondaryIndexBuilder builder)
        Is not scheduled, because it is performing disjoint work from sstable compaction.

      • runIndexSummaryRedistribution

        public java.util.List<SSTableReader> runIndexSummaryRedistribution​(IndexSummaryRedistribution redistribution)
                                                            throws java.io.IOException
        Throws:
        java.io.IOException

      • getDefaultGcBefore

        public static int getDefaultGcBefore​(ColumnFamilyStore cfs,
                                     int nowInSec)

      • getActiveCompactions

        public int getActiveCompactions()

      • isCompactor

        public static boolean isCompactor​(java.lang.Thread thread)

      • incrementAborted

        public void incrementAborted()

      • incrementCompactionsReduced

        public void incrementCompactionsReduced()

      • incrementSstablesDropppedFromCompactions

        public void incrementSstablesDropppedFromCompactions​(long num)

      • getTotalBytesCompacted

        public long getTotalBytesCompacted()

      • getTotalCompactionsCompleted

        public long getTotalCompactionsCompleted()

      • getPendingTasks

        public int getPendingTasks()

      • getCompletedTasks

        public long getCompletedTasks()

      • stopCompaction

        public void stopCompaction​(java.lang.String type)
        Description copied from interface: CompactionManagerMBean
        Stop all running compaction-like tasks having the provided type.
        Specified by:
        stopCompaction in interface CompactionManagerMBean
        Parameters:
        type - the type of compaction to stop. Can be one of: - COMPACTION - VALIDATION - CLEANUP - SCRUB - INDEX_BUILD

      • stopCompactionById

        public void stopCompactionById​(java.lang.String compactionId)
        Description copied from interface: CompactionManagerMBean
        Stop an individual running compaction using the compactionId.
        Specified by:
        stopCompactionById in interface CompactionManagerMBean
        Parameters:
        compactionId - Compaction ID of compaction to stop. Such IDs can be found in the transaction log files whose name starts with compaction_, located in the table transactions folder.

      • setConcurrentCompactors

        public void setConcurrentCompactors​(int value)

      • setConcurrentValidations

        public void setConcurrentValidations()

      • setConcurrentViewBuilders

        public void setConcurrentViewBuilders​(int value)

      • setCoreCompactorThreads

        public void setCoreCompactorThreads​(int number)
        Description copied from interface: CompactionManagerMBean
        Allows user to resize maximum size of the compaction thread pool.
        Specified by:
        setCoreCompactorThreads in interface CompactionManagerMBean
        Parameters:
        number - New maximum of compaction threads

      • setCoreViewBuildThreads

        public void setCoreViewBuildThreads​(int number)
        Description copied from interface: CompactionManagerMBean
        Allows user to resize maximum size of the view build thread pool.
        Specified by:
        setCoreViewBuildThreads in interface CompactionManagerMBean
        Parameters:
        number - New maximum of view build threads

      • interruptCompactionFor

        public void interruptCompactionFor​(java.lang.Iterable<TableMetadata> columnFamilies,
                                   java.util.function.Predicate<SSTableReader> sstablePredicate,
                                   boolean interruptValidation)
        Try to stop all of the compactions for given ColumnFamilies. Note that this method does not wait for all compactions to finish; you'll need to loop against isCompacting if you want that behavior.
        Parameters:
        columnFamilies - The ColumnFamilies to try to stop compaction upon.
        sstablePredicate - the sstable predicate to match on
        interruptValidation - true if validation operations for repair should also be interrupted

      • interruptCompactionForCFs

        public void interruptCompactionForCFs​(java.lang.Iterable<ColumnFamilyStore> cfss,
                                      java.util.function.Predicate<SSTableReader> sstablePredicate,
                                      boolean interruptValidation)

      • waitForCessation

        public void waitForCessation​(java.lang.Iterable<ColumnFamilyStore> cfss,
                             java.util.function.Predicate<SSTableReader> sstablePredicate)

      • getSSTableTasks

        public java.util.List<CompactionInfo> getSSTableTasks()

      • isGlobalCompactionPaused

        public boolean isGlobalCompactionPaused()
        Return whether "global" compactions should be paused, used by ColumnFamilyStore#runWithCompactionsDisabled a global compaction is one that includes several/all tables, currently only IndexSummaryBuilder