General-purpose computers are widely employed across diverse domains owing to their ease of programming. However, they face significant overheads due to extended instruction pipelines necessary for supporting generalization. This has given rise to domain-specific
(a) V20&V21 (b) V21&V23
Figure 1: The distance of the best design found by Au-toDSE [28] between different toolchains: The horizontal axis represents different programs, and the vertical axis denotes the distance between designs. V20: Vitis HLS 2020.2, V21: Vitis HLS 2021.1, V23: Vitis HLS 2023.2. The distance is calculated by summing up the discrete code difference between two designs over each pragma.
accelerators (DSAs) [8,9,14]. However, DSAs pose challenges in programming, restricting their user base to hardware experts [8].
High-level synthesis (HLS) [11] was introduced to simplify the design process by raising the abstraction level from the registertransfer level (RTL) to C/C++. Designers use compiler directives, in the form of pragmas, to describe microarchitecture. While HLS reduces design turn-around times, not every HLS design yields a high-performance microarchitecture [28]. As a result, a new line of research aims to explore the solution space defined by pragmas more efficiently. Because evaluating each design candidate with HLS tools is time-consuming, recent studies propose supervised learning models as proxies of HLS tools, which can predict the quality of results (QoR) to expedite exploration [6,26,[30][31][32]. Training such models involves collecting extensive datasets of diverse designs synthesized by HLS tools. A significant issue arises from the continuous evolution of HLS tools, which can significantly impact QoR values [27]. To gain insight into this issue, we utilized AutoDSE [28] to gather data on different Vitis HLS toolchains: 2020.2 (V20) [1], 2021.1 (V21) [2] and 2023.2 (V23) [3]. Figure 2 illustrates how the performance and resource utilization of each design may vary when changing the toolchain. Additionally, as shown in Table 1, the validity of a design may also fluctuate. During HLS design synthesis, encountering errors is common. For instance, the HLS tool may refuse to synthesize a design with high parallelization factors [26], or it may timeout during synthesis. It is worth noting that we only consider designs within the intersection of those explored by AutoDSE, and more significant changes may occur when evaluating designs across the entire design space. One question that arises is whether changes in performance/resource labels and validity lead to alterations in the optimal design scheme. For instance, a design that previously performed well with one toolchain may become invalid with another. To understand how changes in HLS tools affect the design scheme, we run AutoDSE on these toolchains and compare the best design Table 1: The validity changes when altering the toolchain.
We compare two pairs of commercial HLS tools: V20&V21, V21&V23. "V1" and "V0" stand for the valid or invalid design in the corresponding toolchain.
(a) V20&V21 V21 V1 V21 V0 V20 V1 1462 295 V20 V0 269 348 (b) V21&V23 V23 V1 V23 V0 V21 V1 3690 120 V21 V0 72 1814
it finds. AutoDSE optimizes design performance based on feedback from the toolchain and can naturally adapt to new toolchains. Therefore, any change in the best design found by AutoDSE is attributed to the change in the downstream toolchain. Upon applying AutoDSE to the new toolchain, we observed an average performance improvement of 2×. Furthermore, we noted that the change in the performance of the best design arises from the modification in the design scheme. Essentially, AutoDSE can discover new and improved designs on the new toolchain. Figure 1 depicts the editing distance of the best designs found by AutoDSE on the new and old toolchains. The pragma edit distance is calculated by summing up the differences between each pragma. For example, if one pragma 𝑃 1 can take values from [1,2,5], and one design takes value 5 while the other takes value 1, then their discretized values (index in the design space) are 3 and 1 respectively. This results in a contribution of 3 -1 = 2 to the total editing distance. A larger editing distance indicates a greater shift in the design scheme.
It is therefore critical to design a task transfer learning scheme that could efficiently adapt to the evolving toolchain. According to the previous observations, we observe two major challenges. (1) The first one is to sample efficiently the design from the design space so that the model transferred to those designs can effectively identify the high-quality designs. (2) Another challenge is to adapt the model to the large label shift. Overall, we want to efficiently uncover good designs on the new toolchain when the label shift is unknown and the design space could be as large as 10 13 as we see in the HLSyn benchmark [5].
Model-based optimization is a widely used technique for efficiently sampling high-quality designs. Previous work has utilized Bayesian Optimization [7,29], where sampling is guided by an acquisition function. However, optimizing the acquisition function in a large discrete space, such as HLS designs, remains challenging. Other studies, for example [20] have adapted active learning to achieve better sample efficiency when transferring to a new technology node by proposing data selection algorithms that select diverse and informative data from existing datasets. Our approach is not limited to selecting data from an existing unlabeled data pool, and could navigate the whole design space. In [26,27,[30][31][32], optimizing HLS designs involves training a proxy model on an offline dataset and then conducting optimization with this proxy model. This offline approach requires time-consuming data collection and suffers from generalizability issues when applied to new domains and tasks.
Transfer learning investigates the generalizability of models to unseen domains and tasks, with representation learning playing a critical role in domain transferability [6,13,17,[33][34][35][36]. Some studies propose learned design space pruning for domain transfer in Bayesian Optimization [4,23]. Unlike domain transfer, which assumes a data distribution shift, we focus on task transfer, assuming a tractable change in the black-box function. Previous works [20,27] have shown better task transfer accuracy by freezing some model parameters.
To enable efficient sampling on an unseen toolchain, we propose a novel model-based explorer designed for discrete optimization spaces. We jointly optimize the model prediction and the importance sampling distribution, allowing the sampling process to leverage knowledge from the previous toolchain and achieve better sample efficiency. Moreover, the sampling distribution can guide model updates, enabling the model to minimize its error on a subset of the design space rather than striving for perfect accuracy across the entire space. We also explore learning task-generalizable representations of HLS designs. Our key observation is that the same input often represents similar micro-architectures, despite potential differences in their implementations across toolchains. Thus, we aim to learn representations of HLS designs that can extract their invariance across various toolchains.
Overall, we make the following contributions:
• We propose a model-based explorer that efficiently searches the whole design space, with better sample efficiency compared with the SoTA approach. • We introduce a novel modeling of HLS designs to capture the invariance between toolchains. • Evaluation on 13 programs transferring to V21 shows that we can improve design performance by 2.38× and 1.2× on average, compared with AutoDSE and HARP, with a sample efficiency of 5.75× and a 2.7× runtime improvement. • Evaluation on the latest toolchain reveals that our design outperforms the best of AutoDSE and HARP by 1.27×.
As with AutoDSE and HARP, we utilize the open-source AMD/Xilinx Merlin Compiler [10] to streamline DSE for HLS. It supports three types of pragmas: PIPELINE, PARALLEL, and TILE. Taking designs with these pragmas as input, it automates the synthesis of on-chip buffers and memory interfaces. The PIPELINE, PARALLEL, and TILE pragmas handle loop pipelining, double buffering, loop unrolling, and loop tiling for parallelization and latency hiding. The Merlin Compiler supports standard pipelining and parallelization of HLS, but also offers additional automation for pragmas like "ar-ray_partition" which can be optimally solved given the parallel and tiling factors. This compiler provides a more compact design space and is used in this work. However, our approach can be directly applied to other HLS tools as well.
The HARP framework [27] is the state-of-the-art model-based approach for predicting the performance and resource utilization of HLS designs. It introduces a hierarchical graph representation to mitigate the over-smoothing issue encountered in graph neural networks. By employing a hierarchical graph, the shortest path of the graph can be substantially reduced. This enables the capture of more global information with a shallow GNN. Furthermore, HARP proposes to model the program and pragma transformations separately, treating pragmas as transformations modeled by Multi-Layer Perceptrons (MLPs). This model design aligns with the nature of HLS designs and results in improved accuracy and DSE performance. It trains a classification model to predict the validity and a regression model to predict the QoR. We follow this convention and denote the classification model as R𝜃 𝑐 and the regression model as R𝜃 𝑟 .
Our primary objective is for the learned model to adapt to a new toolchain while identifying Pareto-optimal points and maximizing sample efficiency. Our approach focuses on employing importance sampling to select high-quality designs and integrating model prediction within the importance sampling algorithm to improve sample efficiency. Figure 3 illustrates an overview of our proposed method, which consists of two main components: the DS Sampler and Reward Model Transfer Learning. Using the Cross-Entropy Method (CEM), the DS Sampler performs importance sampling based on the design performance. It iteratively optimizes a probability distribution across the entire design space, concentrating higher density on designs with high rewards. This approach ensures that high-reward designs are sampled more frequently, thereby increasing the likelihood of identifying optimal designs. To further boost sample efficiency, we introduce a reward model that predicts the validity and QoR of HLS designs. This model is adapted from previous toolchains, leveraging data collected from earlier implementations to enhance prediction accuracy. To address the potential label shift due to toolchain changes and distribution shift from the importance sampling, we integrate an active learning subroutine within the CEM algorithm. In summary, we facilitate an efficient transfer to a new toolchain through a model-based method, which involves two iterative steps: fitting the model to the current sampling distribution and updating the sampling distribution based on the model's recommendations.
In Section 3.1, we will describe version-invariant modeling that efficiently adapts the model to the label shift. In Section 4, we will explain the model-based importance sampling algorithm based on CEM.
A critical problem of training a prediction model for HLS designs is how to train a robust model when the labeled data is scarce compared to the size of the design space. One way to mitigate the insufficiency of labeled data is to transfer knowledge from all existing data, possibly collected from different toolchains. Our intuition is to model a robust representation of HLS designs that contains the invariant information between different toolchains. In practice, such invariance is abundant. For example, the same input program with the same pragmas represents similar microarchitectures, and similar microarchitectures will not result in completely different implementations. Following this intuition, we consider learning invariant embedding for task transfer on the HLS design by training a single model with data from different toolchains.
One problem to tackle is to train a model that is aware of the version of the data. A simple solution is to include the version tag in the input data. However, when switching to a new version, we may need a completely new tag, and that will cause a huge shift in input data. Another approach is to have a shared encoder, but different decoder for each task. Each decoder is trained on data from its corresponding toolchain, allowing the unique parameters of the decoders to capture the specific characteristics of each toolchain. This approach follows the intuition that the same input HLS design represents a similar micro-architecture for different toolchains, and could be represented as the same embedding in the latent space. However, we argue that it is not enough to just model the shared information between toolchains. A shared embedding may not contain enough information to address the variance between toolchains. In HLS design, the information that remains completely invariant is the program. But for pragma transformations, it remains unknown exactly what implementation stays the same and what is changed. Following this intuition, we propose the following modeling scheme:
Separate embedding for shared and private information. Fig. 4 shows the model architecture. For each input design, two separate embeddings of the design will be learned. One of the embeddings is modeled by a set of parameters shared among all toolchains, and the other embedding is modeled with a set of parameters that is private to each toolchain. Then, we concatenate the two embeddings and forward it to different decoders for each toolchain.
Another benefit of the proposed model architecture is that it can be trained jointly with data from multiple versions. We optimize the mean-squared error (MSE) averaged over all versions of data. Intuitively, training with more data will help the model get a stronger representation of the invariant part. Moreover, once diverse data from different toolchains have been gathered, the invariant part R 𝑡 𝑠 ← Evaluate 𝑅(𝑥) for 𝑥 ∈ D 𝑡 𝑠 .
8:
𝑞 ← Calculate the 1 -𝜌 quantile of R 𝑡 𝑠 10:
11:
𝑝 (𝑥) = (1 -𝛼)𝑝 (𝑥) + 𝛼 p (𝑥) 12: end for 13: return 𝑟 max can be pretrained, and we are able to fix the invariant part and fine-tune only the private part when switching to a new toolchain. This will result in fine-tuning fewer parameters when transferring to a new toolchain and better efficiency.
We will now describe the original model-free version of the Cross-Entropy Method (CEM) [15] and how it can be applied to optimize HLS design. It was originally designed for rare-event simulation but can be applied for global optimization, and it has the ability to converge to global optimum theoretically [22,24]. While the cross-entropy method is treated as a model-based technique in some work [12] because it contains a model for the distribution of rare events (good designs), we treat the original version of the algorithm as model-free since it does not contain a learned model of the reward function. We will show that the cross-entropy method is a strong candidate for efficient global optimization.
Algorithm 1 outlines the application of the Cross-Entropy Method (CEM) for optimizing black-box functions. The process begins by initializing the sampling distribution 𝑝 (𝑥) to a random distribution (line 3). In each iteration, 𝑁 samples are drawn from the current distribution and evaluated using the ground truth function 𝑅 (lines 6, 7). Based on these evaluations, the distribution is then updated to concentrate more on samples that exceed the (1 -𝜌) quantile of the 𝑁 rewards, i.e. focusing on the top 𝑁 • 𝜌 performing samples (lines 9-11). To maintain randomness and prevent overfitting, the distribution is modified by a step size 𝛼, which typically ranges from 0.7 to 0.9. This algorithm is noted for its rapid convergence rate and robustness against different hyperparameters, as introduced in [15,21].
To tailor Algorithm 1 for HLS design optimization, the key tasks involve modeling the distribution of design configurations and implementing updates to this distribution. In HLS, each design is characterized by a combination of different pragma values. Each pragma within the program must either be assigned a specific value or be set to a default value that effectively disables it, thereby completing the design configuration. We model the distribution over all possible designs by assigning a probability vector 𝑝 𝑖 (𝑥) for each pragma 𝑖, resulting in 𝑝 (𝑥) = Π 𝑖 𝑝 𝑖 (𝑥 𝑖 ). The independent (a) CEM-5000 (b) CEM-10000 Figure 6: The convergence rate of CEM-K. The horizontal axis shows the number of iterations, and the vertical axis depicts the logarithmic speedup. The different lines illustrate different kernels. The performance typically reaches a plateau in less than 10 iterations.
modeling reduces the likelihood of overfitting when the number of data points available for distribution calculation is limited. We will demonstrate that this method of distribution modeling not only simplifies sampling distribution updates but also yields effective optimization results for HLS designs.
To implement the distribution update, we first get the one-hot embedding of each design. Specifically, for each design 𝑥, we use a one-hot vector 𝑥 𝑖 for each pragma, where 𝑥 𝑖 𝑗 = 1 if the discretized value of pragma 𝑖 is 𝑗, and 𝑥 𝑖 𝑗 = 0 otherwise. For example, consider a design space containing two pragmas 𝑃 𝑎 and 𝑃 𝑏 , and suppose 𝑃 𝑎 can take value from [1, 2, 5] and 𝑃 𝑏 can take value from [1,2]. Then the design 𝑥 = {𝑃 𝑎 = 5, 𝑃 𝑏 = 2} will be encoded with
Then, at each iteration, the distribution p in Algorithm 1 can be calculated by separately calculating the probability vector 𝑝 𝑖 of each pragma 𝑖, with the one-hot vector 𝑥 𝑖 :
where 𝑅 is the ground truth reward function, D 𝑡 𝑠 denotes the 𝑁 samples generated by the previous distribution 𝑝 at iteration 𝑡, and the 𝑞 is the 1 -𝜌 quantile of 𝐺 𝑟 .
When implementing the algorithm, we also memorize the best 𝜌 • 𝑁 design in all previous iterations and consider them when calculating the quantile and optimizing the distribution, which improves the performance. This detail is omitted in the algorithm description for simplicity.
We first conduct experiments to validate that CEM is good for global optimization, comparing it with other algorithms including breadth-first search (BFS), simulated annealing, and genetic algorithm. Due to the long runtime of evaluating HLS design, we use the model prediction as the ground truth function. Even though high predicted performance may not result in high actual performance, it is a good indicator of the ability of the optimizer. We evaluate 5 programs with a large design space and calculate the geomean of the top-1 latency under resource constraints. We also test CEM and genetic search with different population sizes. Fig. 5 displays the normalized result with respect to BFS. Gen-𝐾 and CEM-𝐾 stand for the genetic algorithm and CEM under different population sizes. Under the same runtime budget, the CEM explorer performs the best among all baselines, and the genetic search also achieves a comparable performance. When the population size is larger, the genetic search will run fewer iterations under the same time limit, resulting in the performance degradation from Gen-500 to Gen-5000. On the other hand, CEM could uncover a better design when increasing the population size, due to its fast convergence rate. As illustrated in Fig. 6, the performance typically converges after 8 iterations. However, the result also shows that optimization with CEM is not sample-efficient. It would require a very large population size, typically larger than 1000, to reach a stable performance. When the population size 𝑁 is small, the distribution may quickly overfit to a specific design. Evaluating more than 1000 designs using HLS is very time-consuming, even if a dozen of designs can be evaluated in parallel. So, we consider inserting model prediction in the loop to improve the sample efficiency of the CEM in the next section.
One simple way to achieve sample efficiency is to replace the evaluation on the true reward function 𝑅 with a proxy model R𝜃 , pick the design with the top prediction value, and validate it with 𝑅 at the last iteration. However, such an approach may cause severe performance degradation, because the model R𝜃 is not accurate in terms of the label shift caused by changing the toolchain, and the distribution shift caused by a sampling distribution update. In Section 1, we illustrate how the model could be vulnerable to the label shift. Another factor to consider is the distribution shift. Even if the model has already adapted to the label shift with data on the new version, it may have low accuracy on some designs visited by the explorer, because of the mismatch between the training data distribution induced by the data gathering heuristic, and the testing distribution induced by the explorer. Therefore, it is possible for the explorer to find a design with a high predicted performance but a low actual performance. Such a phenomenon is constantly observed when trying to solely rely on the proxy model to do the optimization [18,19]. Intuitively, a powerful optimization algorithm will put pressure on the model to be perfectly accurate on the whole design space, which is usually hard to achieve given limited labeled data.
Therefore, it is critical to first update the model to have a reasonably accurate prediction under the current function 𝑅(𝑥) and the current data distribution 𝑝 (𝑥). Typically, we want to optimize the following objective, where D is the whole design space.
When the sampling distribution updates, the objective becomes a weighted error, assigning greater weight to regions that potentially contain high-quality designs. Although it is still sample inefficient to minimize the error of the model with arbitrary data distribution on the whole design space, we find that it is possible to achieve better sample efficiency by integrating the model update into the optimization process. For the CEM, if the model can have an accurate prediction on D 𝑡 𝑠 in each iteration, then the next distribution can be calculated solely based on the model prediction. Note that D 𝑡 𝑠 ⊂ D is sampled from the distribution 𝑝 (𝑥), and is an unlabeled pool of the design. As discussed in section 4.1, the size of D 𝑡 𝑠 could still be large, so we want to further improve the sample efficiency, by selecting several designs from D 𝑡 𝑠 to update the model. The sub-problem to solve then is: How to select a few designs in D 𝑡 𝑠 and update the model on the true value of these designs, so that the model has a minimal error on D 𝑡 𝑠 . More specifically, we want to find D 𝑡 𝑐𝑜𝑟𝑒 ⊂ D 𝑡 𝑠 , so that the model can have minimal error on D 𝑡 𝑠 when the parameter 𝜃 is updated on D 𝑡 𝑐𝑜𝑟𝑒 . 4.2.1 Selective Labeling for Model Update. Several heuristics have been proposed to tackle the data selection problem in active learning. Those heuristics primarily consider two objectives, uncertainty and diversity. Both objectives are important to minimize the model error given a fixed labeling budget.
However, the uncertainty-based approaches can only characterize the distribution shift, the change of 𝑝 (𝑥). In task transfer, it is also important to be aware of the change in the function 𝑅(𝑥), or 𝑝 (𝑦|𝑥). While it is hard to estimate the change without actually observing the true value, it is still reasonable to sample diverse designs under a fixed budget, so that more information can be revealed about the unknown function. Therefore, we choose to use the coreset approach [25] to acquire diverse data.
The coreset based sampling algorithm approximates the model error on the whole pool of data with distance measurement from selected data points and the rest of the data. [25] and [20] separately model the problem as K-Center and K-Means clustering. We consider utilizing the K-Means algorithm on the hidden representation due to its efficiency. Since we have both a regression model R𝜃 𝑟 to predict the QoR and a classification model R𝜃 𝑐 to predict the validity of the design, we need samples to provide information for both models. We thus divide the 𝐾 labeling budget into 𝐾 1 and 𝐾 0 and divide the unlabeled pool D 𝑡 𝑠 into two subsets D 𝑡 𝑠 1 and D 𝑡 𝑠 0 . D 𝑡 𝑠 1 contains designs that the classification model R𝜃 𝑐 predicts valid, and D 𝑡 𝑠 0 contains those that the model predicts invalid. Then, we run K-Means to get 𝐾 1 cluster centroids of the hidden embeddings of R𝜃 𝑟 on D 𝑡 𝑠 1 and 𝐾 0 cluster centroids of the hidden embeddings of R𝜃 𝑐 on D 𝑡 𝑠 0 . Then, the designs closest to the cluster centroids are selected to be added to the database. 4.2.2 Active-CEM. We now describe the complete exploration algorithm which consists of an outer loop that updates the sampling distribution according to the model prediction, and an inner loop that fits the model to the current distribution. We abbreviate the name with Active-CEM.
As shown in Algorithm 2, the sampling distribution 𝑝 is initialized to random, and the model R𝜃 𝑟 , R𝜃 𝑐 is initialized with the old model trained on the old database. A database 𝐵 that has designs with true labels is initialized to empty. For each iteration 𝑡, an unlabeled pool D 𝑡 𝑠 will be sampled according to distribution 𝑝. Then, the unlabeled pool D 𝑡 𝑠 , the database 𝐵, and the current model will be forwarded to the inner active learning loop with the K-Means
Algorithm 2 Active-CEM Input: num iterations 𝑛, population size 𝑁 , true function 𝑅, quantile 𝜌, old model R′ 𝜃 𝑟 , R′ 𝜃 𝑐 , active learning round 𝑚, batch size 𝑏 Output: best design 𝑑 best , labeled database 𝐵 Initialize distribution 𝑝 (𝑥) to random. R𝜃 𝑟 , R𝜃 𝑐 ← R′ 𝜃 𝑟 , R′ 𝜃 𝑐 for 𝑡 = 1 to 𝑛 do D 𝑡 𝑠 ← 𝑁 samples without replacement from 𝑝 (𝑥) # actively update model for 𝑗 = 1 to 𝑚 do D 𝑡 𝑐𝑜𝑟𝑒 ← select 𝑏 samples from D 𝑡 𝑠 # K-Means 𝐵 = 𝐵 ∪ (D 𝑡 𝑐𝑜𝑟𝑒 , 𝑅(D 𝑡 𝑐𝑜𝑟𝑒 )) # run HLS on each instance in D 𝑡 𝑐𝑜𝑟𝑒 , add to 𝐵 R𝜃 𝑟 , R𝜃 𝑐 ← train_model(𝐵, R𝜃 𝑟 , R𝜃 𝑐 ) end for # use model prediction to update sampling distribution # invalid points assigned to -∞ R 𝑡 𝑠 (𝑥) = R𝜃 𝑟 (𝑥) + ∞ • ( R𝜃 𝑐 (𝑥) -1) for 𝑥 ∈ D 𝑡 𝑠 𝑞 ← Calculate the 1 -𝜌 quantile of R 𝑡 𝑠 Update 𝑝 (𝑥) according to R 𝑡 𝑠 and 𝑞 end for return best design found, 𝐵, ( R𝜃 𝑟 , R𝜃 𝑐 )
algorithm. The active learning loop will return an updated model. Then, the updated model will predict the label for all designs in D 𝑡 𝑠 . According to the predicted label, the 1 -𝜌 quantile of the design performance will be calculated on the valid designs that satisfy the resource constraint, and the distribution will be updated according to Equation 1. The only difference is that the true value 𝑅(𝑥) in equation 1 is replaced with the updated prediction model R𝜃 𝑟 (𝑥) and R𝜃 𝑐 (𝑥). 4.2.3 Sample efficiency and runtime of the algorithm. We briefly discuss the sample efficiency and the runtime of the proposed algorithm. By sample efficiency, we mean the number of designs evaluated with the HLS toolchain during exploration, which is the major runtime bottleneck. As shown in algorithm 2, the modelfree version of the CEM algorithm needs to sample 𝑁 • 𝑛 design points, while the model-based version only requires 𝑚 • 𝑏 • 𝑛. As for the runtime, since the design is evaluated in batch, the runtime is bounded by the number of parallel HLS workers and the number of samples. The overhead of sampling unlabeled design and updating the distribution are negligible. Suppose there are 𝐶 parallel workers to run HLS, and the timeout for running HLS design is 𝑇 , then the runtime of the model-free version is 𝑇 𝑛 • 𝑁 𝐶 , while the runtime of the model-based version is 𝑇 𝑛𝑚 • 𝑏 𝐶 . While proper scheduling could reduce the overall runtime of evaluating 𝑁 design with 𝐶 workers, we provide an upper bound of the runtime here.
Another way to achieve sample efficiency is to prune away part of the large design space so that the explorer will have less pressure to explore a smaller design space. While design space pruning is not the focus of this work, we empirically find that the "TILE" pragma is not as efficient as the "PARALLEL" and "PIPELINE" pragma for optimizing design. Also, whether or not "TILE" pragmas would be effective is determined mostly by the program itself (whether it is memory bound/compute bound), and is more agnostic to the change in the toolchain. Therefore, we consider pruning away "TILE" pragmas for some programs. We adopt a conservative heuristic that prunes away "TILE" pragmas only if (1) less than 10% of the designs in the old database have tiling pragma, (2) the top-10 designs do not contain "TILE" pragma, and (3) the memory footprint of the workload could fit on-chip. We only cut the "TILE" pragma if all these conditions are satisfied. With this heuristic, the model will also witness less distribution shift at the early stage of exploration.
We evaluate our approach on two task transfer scenarios: from AMD Xilinx Vitis 2020.2 (V20) to Vitis 2021.1 (V21) and from Vitis 2021.1 (V21) to Vitis 2023.2 (V23). We also utilize the AMD Xilinx Merlin Compiler [10]. We assess our methods on 13 programs from the HLSyn [5] dataset, from toolchain V20 to V21. We do not consider programs that have a very small design space. The workload covers dense linear algebra, data analytics, and stencil operations.
To evaluate the robustness of the method to different label shifts, we further test it by transferring from V21 to V23, on 5 programs with very large design space. We list the domain and the size of the design space for each program in Table 2. "MM" stands for matrix-matrix multiplication, and "MV" stands for matrix-vector multiplication. Since we focus on assessing task transfer to new toolchains, we fixed the evaluation platform to Xilinx U200 with a frequency of 250MHz. While the proposed model architecture introduces extra overhead, we simply finetune the HARP model during the CEM exploration. After collecting a database with the explorer, we train the new model architecture with invariant embedding on the database and conduct one last round of DSE with a BFS search. The model is trained with a batch size of 64 and is trained for 1500 epochs, and the learning rate is set to 1e-3 with a linear warmup and is cosine annealed to 1e-5. When training the model with invariant embedding, we randomly sample mini-batches of 64 from the V18 and the V20 database, and forward all three batches in a single iteration to calculate the loss. We run each experiment three times with different random seeds, under the same split of the dataset, and report the mean and standard deviation of each method.
We validate the effectiveness of learning invariant embedding on the HLS designs with model accuracy. We compare the proposed model (Inv-Hybrid) with three baselines, the HARP model trained from scratch (HARP scratch) on the new version data, the HARP model initialized with pre-trained parameters on old data and fine-tuned on new data (HARP finetune). For the old version data, we utilize the HLSyn database, which consists of data collected by AutoDSE on V18 and V20 of the toolchain. For the new version data, we utilize the data collected by AutoDSE on the V21 toolchain for the 13 programs we selected, following the data collection pipeline as described in HLSyn. 3975 regression data is gathered for the 13 programs on the V21 toolchain. And the V18 and V20 databases contain 9439 and 5194 data. We split the V21 data into training, validation, and test set, with a portion of 40%, 10%, and 50%, to mimic the scenario where a few data is gathered on the large design space. We select the model with a minimum validation error and record the testing accuracy. Table 3 summarizes the efficiency of each model, using the regression RMSE as a metric. We observe that the HARP model performs much better when initialized with pre-trained parameters, which is consistent with the observation made previously in [27]. With the Inv-Hybrid model, the RMSE on the test set further improves by 11% and shows better stability during training with less standard deviation. This validates our assumption that we should model both the invariant and the difference between toolchains to achieve good transfer learning accuracy.
We next verify the efficiency of the Active-CEM algorithm. We compare with AutoDSE and HARP. For the AutoDSE baseline, we follow previous work and run all three explorers in parallel. Following [27], we set a timeout of 24 hours and do not terminate the jobs that are already running, resulting in a maximum runtime of 25 hours. For baseline evaluation, we adopt the same approach as HARP for running its explorer. Specifically, after collecting data with Au-toDSE, we initialize the model with the pre-trained parameter on the old version, and then fine-tune the HARP model on the new version data. Then we run a BFS search for 1 hour with HARP and validate the design with top-10 predicted performance using the HLS toolchain. When transferring to V21, we initialize the model with parameters pre-trained on the V20 database. When transferring to V23, we initialize the model with parameters pre-trained on the V21 database. For a fair comparison, when pretraining the model on existing versions of databases, we only utilize the database collected by AutoDSE. For Active-CEM, similar to HARP, we initialize the model pre-trained parameter on the old version data. For each program, we run Active-CEM for 8 iterations to converge, 4 and Table 5 illustrate the result of the design performance when transferring from V20 to V21 and from V21 to V23. When transferring to V21, exploration with Active-CEM performs 2.34× better than AutoDSE and 1.18× better than the best of AutoDSE and HARP and achieves this with a 5.75× total sample efficiency and a 1.74× sample efficiency in terms of valid data. Note that Active-CEM tends to explore more valid designs during policy updates while AutoDSE does not contain heuristics to explicitly sample valid designs, therefore a larger portion of the designs explored by AutoDSE are invalid. As for design space pruning, among all 13 programs, only the design spaces of 3 programs (gemver, 3mm, and fdtd-2d) are pruned without searching for "tile" pragmas. We leave learning a more robust design space pruning strategy to future work. When transferring to V23, Active-CEM still performs 1.3× better than AutoDSE, and 1.27× better than the best of AutoDSE and HARP, and is a 8.85× sample efficiency in total and a 3.47× sample efficiency in terms of valid design, demonstrating the robustness of the proposed method. Better design performance and better sample efficiency are achieved by inserting model prediction into an importance sampling method, utilizing the knowledge learned on the old toolchains and addressing the label and distribution shift.
We further train the HARP model and the Invariant-hybrid model with the database collected by the CEM method, and conduct one last round of DSE with a BFS search. Our result in Table 4 reveals that we can locate a better design in one last round of the exploration, achieving another moderate 1.01𝑥 speedup on average.
We further perform an ablation study by running AutoDSE for the same amount of time as Active-CEM. Specifically, we pick the checkpoint of AutoDSE at 10 hours and report the best design found by all three explorers. As seen in Table 5, Active-CEM outperforms AutoDSE on average by 1.39× when the runtime budget is the same. 6 shows the result of transferring the model by fixing the first several layers of the GNN encoder. Similar to the previous experiment, we split the data with 40%, 10%, and 50%, conduct each experiment 3 times with different random seeds, and report the mean and standard deviation. We observe that the model achieves the best accuracy when we do not freeze any parameter, which is different from the conclusion drawn from various transfer learning works with image data [20]. We hypothesize that this is because of the large label shift caused by switching to a different toolchain, and the lack of proper pretraining on a large database that contains multiple versions of data.
5.4.2 Separate modeling the invariance and the difference. We add another baseline to confirm the effectiveness of the proposed model architecture, which models the invariance and the difference between toolchains. Specifically, we implement a model architecture as shown in Fig. 7, which simply uses a different decoder to decode a shared embedding to different outputs. Moreover, we fix the parameter size of both architectures to be the same, so that we can strictly study the effect of decoupling the shared and the private embedding. In Table 7, the test RMSE of "Inv-Hybrid" is lower than "Inv-Share" by 11%, demonstrating the importance of modeling both the invariance and the difference between toolchains. 5.4.3 Design space pruning. We evaluate the efficiency of design space pruning by comparing the design performance when enabling and disabling it. Only 3 out of 13 programs satisfy the pruning constraints and we report the performance difference for them. Table 8 illustrates how the design performance improves when design space pruning is enabled.
In this work, we discuss the sample efficiency challenge in the task transfer learning problem for HLS DSE. We mitigate the challenge with a novel approach on both sampling and modeling. On sampling, we propose Active-CEM, a model-based explorer that efficiently samples high-quality designs in the whole design space.
On modeling, we propose to model both the invariance and difference between toolchains and jointly train a single model with data collected from multiple toolchains. Moving forward, we plan to evaluate our sampling method in the domain-transfer setting and consider task-transfer learning for different devices. We also plan to combine domain knowledge in the sampling process for better efficiency. In [16], we extend our task transfer results to the complete HLSyn dataset on V21 toolchain.